| New file |
| | |
| | | # kb-cli 对标 CodeGraph 增强 实施计划 |
| | | |
| | | > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| | | |
| | | **Goal:** 按 spec `docs/superpowers/specs/2026-09-03-kb-cli-codegraph-alignment-design.md` 实现 kb-cli 的 RWR 图排序、CJK 混合检索、增量同步、悬空双链自动补全、explore 命令。 |
| | | |
| | | **Architecture:** SQLite(WAL) 单库存储,schema v2 迁移加列(aliases/status/provenance/文件指纹);FTS5 改 external-content + 触发器;搜索 = FTS(ASCII 词) + LIKE(CJK 词) 双通道 → RWR 图质量 → min-max 双信号加权;索引改为 (size, mtime, sha256) 对账增量同步;wikilink 解析失败入 unresolved_links 表,新节点入库时自动重试。 |
| | | |
| | | **Tech Stack:** Go 1.22.2, mattn/go-sqlite3 (FTS5), cobra, yaml.v3 |
| | | |
| | | ## Global Constraints |
| | | |
| | | - **构建标志(铁律)**:`CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm"`——本机 gcc 下 go-sqlite3 自带 amalgamation 链接 libm 失败导致 FTS5 编译失败;系统 libsqlite3 (3.45.1) 原生含 FTS5,加此两参数后全部测试通过。已存于 `go.env`,Makefile 必须固化 |
| | | - 不新增第三方依赖(stdlib + go.mod 现有依赖) |
| | | - 注释/错误信息用中文,与现有代码风格一致 |
| | | - 每个任务 TDD:先写失败测试 → 跑失败 → 最小实现 → 跑通过 → commit |
| | | - 测试命令模板:`CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./... 2>&1 | grep -v "no test files"` |
| | | - 现有代码约定:`nodes.path` UNIQUE;虚拟节点(tag/entity)ID 从 1000000 起;边表 `UNIQUE(from_node, to_node, relation, label)` |
| | | - spec 全文在 `docs/superpowers/specs/2026-09-03-kb-cli-codegraph-alignment-design.md`,本计划与其冲突时以 spec 为准 |
| | | |
| | | **实测背景(计划依据):** |
| | | - 真实库 `~/rag-lpg-obsidian`(458 个 .md)上 `search 补气` 只返回 1 条——FTS5 unicode61 把连续中文当整串单 token,`MATCH '补气'` 匹配不到"电子秤补气失败";而 `LIKE '%补气%'` 命中 16 条 content + 1 条 title。**CJK 词必须走 LIKE 通道** |
| | | - trigram tokenizer 实测:多字符 CJK 词部分可用("红绿闪"命中)但单字符("秤")不命中,且改变现有 FTS 行为面太大——不采用,保持 unicode61 |
| | | |
| | | --- |
| | | |
| | | ### Task 1: 构建环境固化 + Schema v2 迁移 |
| | | |
| | | **Files:** |
| | | - Create: `internal/index/migrations.go` |
| | | - Create: `internal/index/migrations_test.go` |
| | | - Modify: `internal/index/sqlite.go:48-87`(initTables) |
| | | - Modify: `internal/index/fts.go:18-37`(CreateFTS/PopulateFTS) |
| | | - Create: `Makefile` |
| | | - Modify: `go.env`(已存在,确认内容) |
| | | |
| | | **Interfaces:** |
| | | - Consumes: 现有 `Store.Open` |
| | | - Produces: `func (s *Store) migrate() error`(Open 时自动调用);v2 后 `nodes` 表含列:`aliases TEXT, status TEXT, size INTEGER, mtime INTEGER, content_hash TEXT`;`edges` 表含列 `provenance TEXT`;新表 `unresolved_links(id, from_node, link_text, name_tail, status, created_at, last_attempt)` 与 `schema_versions`;`nodes_fts` 为 external-content 虚拟表(title, content, tags, aliases)带增删改触发器 |
| | | |
| | | - [ ] **Step 1: 固化构建标志(Makefile)** |
| | | |
| | | ```makefile |
| | | # Makefile |
| | | CGO_CFLAGS := -DSQLITE_ENABLE_FTS5 |
| | | CGO_LDFLAGS := -lm |
| | | |
| | | .PHONY: build test clean |
| | | |
| | | build: |
| | | CGO_CFLAGS="$(CGO_CFLAGS)" CGO_LDFLAGS="$(CGO_LDFLAGS)" go build -o kb-cli . |
| | | |
| | | test: |
| | | CGO_CFLAGS="$(CGO_CFLAGS)" CGO_LDFLAGS="$(CGO_LDFLAGS)" go test ./... 2>&1 | grep -v "no test files" |
| | | |
| | | clean: |
| | | rm -f kb-cli |
| | | ``` |
| | | |
| | | - [ ] **Step 2: 写迁移失败测试** |
| | | |
| | | `internal/index/migrations_test.go`: |
| | | |
| | | ```go |
| | | package index |
| | | |
| | | import ( |
| | | "os" |
| | | "path/filepath" |
| | | "testing" |
| | | ) |
| | | |
| | | // newV1Store 手工构建一个 v1 schema 的老库(模拟迁移前状态) |
| | | func newV1Store(t *testing.T) (*Store, string) { |
| | | t.Helper() |
| | | dbPath := filepath.Join(t.TempDir(), "kb-v1.db") |
| | | db, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL") |
| | | if err != nil { |
| | | t.Fatal(err) |
| | | } |
| | | schema := ` |
| | | CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL); |
| | | CREATE TABLE nodes ( |
| | | id INTEGER PRIMARY KEY AUTOINCREMENT, |
| | | path TEXT NOT NULL UNIQUE, |
| | | title TEXT, section TEXT, tags TEXT, entities TEXT, wikilinks TEXT, |
| | | content_fts TEXT, |
| | | created_at TEXT DEFAULT (datetime('now')), |
| | | updated_at TEXT DEFAULT (datetime('now'))); |
| | | CREATE TABLE edges ( |
| | | id INTEGER PRIMARY KEY AUTOINCREMENT, |
| | | from_node INTEGER NOT NULL, to_node INTEGER NOT NULL, |
| | | relation TEXT NOT NULL, label TEXT, |
| | | UNIQUE(from_node, to_node, relation, label)); |
| | | ` |
| | | if _, err := db.Exec(schema); err != nil { |
| | | t.Fatal(err) |
| | | } |
| | | if _, err := db.Exec(`INSERT INTO nodes (path, title, tags, entities, wikilinks, content_fts) |
| | | VALUES ('FAQ/001-测试.md', '测试', '["a"]', '[]', '[]', '内容')`); err != nil { |
| | | t.Fatal(err) |
| | | } |
| | | return &Store{db: db}, dbPath |
| | | } |
| | | |
| | | func TestMigrateV1ToV2(t *testing.T) { |
| | | store, dbPath := newV1Store(t) |
| | | defer store.Close() |
| | | |
| | | // 迁移前备份文件必须存在 |
| | | if err := store.migrate(); err != nil { |
| | | t.Fatalf("migrate: %v", err) |
| | | } |
| | | if _, err := os.Stat(dbPath + ".bak"); err != nil { |
| | | t.Fatalf("备份文件不存在: %v", err) |
| | | } |
| | | |
| | | // v2 列必须存在 |
| | | cols := store.nodeColumns(t) |
| | | for _, want := range []string{"aliases", "status", "size", "mtime", "content_hash"} { |
| | | if !cols[want] { |
| | | t.Errorf("nodes 缺列 %s", want) |
| | | } |
| | | } |
| | | if !store.edgeHasProvenance(t) { |
| | | t.Error("edges 缺 provenance 列") |
| | | } |
| | | if !store.tableExists(t, "unresolved_links") { |
| | | t.Error("缺 unresolved_links 表") |
| | | } |
| | | |
| | | // 数据保留 |
| | | var title string |
| | | if err := store.db.QueryRow(`SELECT title FROM nodes WHERE path='FAQ/001-测试.md'`).Scan(&title); err != nil { |
| | | t.Fatal(err) |
| | | } |
| | | if title != "测试" { |
| | | t.Errorf("迁移丢数据: %s", title) |
| | | } |
| | | } |
| | | |
| | | func TestMigrateIdempotent(t *testing.T) { |
| | | store, _ := newV1Store(t) |
| | | defer store.Close() |
| | | if err := store.migrate(); err != nil { |
| | | t.Fatalf("第一次: %v", err) |
| | | } |
| | | if err := store.migrate(); err != nil { |
| | | t.Fatalf("第二次(幂等): %v", err) |
| | | } |
| | | } |
| | | |
| | | func TestNewDBIsV2(t *testing.T) { |
| | | dbPath := filepath.Join(t.TempDir(), "kb-v2.db") |
| | | store, err := Open(dbPath) |
| | | if err != nil { |
| | | t.Fatal(err) |
| | | } |
| | | defer store.Close() |
| | | if !store.edgeHasProvenance(t) { |
| | | t.Error("新建库应为 v2 schema") |
| | | } |
| | | } |
| | | |
| | | // 测试辅助:列检查 |
| | | func (s *Store) nodeColumns(t *testing.T) map[string]bool { |
| | | t.Helper() |
| | | rows, err := s.db.Query(`SELECT name FROM pragma_table_info('nodes')`) |
| | | if err != nil { |
| | | t.Fatal(err) |
| | | } |
| | | defer rows.Close() |
| | | m := map[string]bool{} |
| | | for rows.Next() { |
| | | var name string |
| | | rows.Scan(&name) |
| | | m[name] = true |
| | | } |
| | | return m |
| | | } |
| | | |
| | | func (s *Store) edgeHasProvenance(t *testing.T) bool { |
| | | t.Helper() |
| | | var n int |
| | | s.db.QueryRow(`SELECT COUNT(*) FROM pragma_table_info('edges') WHERE name='provenance'`).Scan(&n) |
| | | return n > 0 |
| | | } |
| | | |
| | | func (s *Store) tableExists(t *testing.T, name string) bool { |
| | | t.Helper() |
| | | var n int |
| | | s.db.QueryRow(`SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name=?`, name).Scan(&n) |
| | | return n > 0 |
| | | } |
| | | ``` |
| | | |
| | | 注意:测试文件顶部需 `import "database/sql"`。 |
| | | |
| | | - [ ] **Step 3: 跑测试确认失败** |
| | | |
| | | Run: `CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./internal/index/ -run TestMigrate -v` |
| | | Expected: FAIL — `store.migrate undefined` |
| | | |
| | | - [ ] **Step 4: 实现 migrations.go** |
| | | |
| | | `internal/index/migrations.go`: |
| | | |
| | | ```go |
| | | package index |
| | | |
| | | import ( |
| | | "fmt" |
| | | "os" |
| | | "path/filepath" |
| | | ) |
| | | |
| | | const currentSchemaVersion = 2 |
| | | |
| | | // migrate 幂等的 schema 升级:v1 → v2。 |
| | | // v2 变更:nodes 加 aliases/status/size/mtime/content_hash 列; |
| | | // edges 加 provenance 列;新建 unresolved_links、schema_versions 表; |
| | | // nodes_fts 改 external-content + 触发器。迁移前自动备份 .db 文件。 |
| | | func (s *Store) migrate() error { |
| | | // 读当前版本 |
| | | var version int |
| | | err := s.db.QueryRow(`SELECT value FROM schema_versions WHERE key='version'`).Scan(&version) |
| | | if err != nil && version == 0 { |
| | | // 首次:schema_versions 表可能不存在 |
| | | version = 1 |
| | | } |
| | | if version >= currentSchemaVersion { |
| | | return nil |
| | | } |
| | | |
| | | // 备份 |
| | | if err := s.backupDB(); err != nil { |
| | | return fmt.Errorf("备份失败: %w", err) |
| | | } |
| | | |
| | | // v1 → v2 |
| | | if err := s.migrateV1toV2(); err != nil { |
| | | return fmt.Errorf("v1→v2 迁移失败: %w", err) |
| | | } |
| | | return nil |
| | | } |
| | | |
| | | func (s *Store) backupDB() error { |
| | | // dbPath 从 DSN 提取:DSN 形如 <path>?_journal_mode=WAL |
| | | for _, row := range s.db.QueryRow(`PRAGMA database_list`).Rows() { |
| | | } |
| | | // 简化:Open 时把 dbPath 存到 Store |
| | | if s.dbPath == "" { |
| | | return nil // 无路径信息时跳过备份(测试场景) |
| | | } |
| | | dst := s.dbPath + ".bak" |
| | | if _, err := os.Stat(dst); err == nil { |
| | | return nil // 已有备份 |
| | | } |
| | | data, err := os.ReadFile(s.dbPath) |
| | | if err != nil { |
| | | return err |
| | | } |
| | | return os.WriteFile(dst, data, 0644) |
| | | } |
| | | |
| | | func (s *Store) migrateV1toV2() error { |
| | | // 1. schema_versions 表 |
| | | if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS schema_versions ( |
| | | key TEXT PRIMARY KEY, value TEXT NOT NULL)`); err != nil { |
| | | return err |
| | | } |
| | | if _, err := s.db.Exec(`INSERT OR REPLACE INTO schema_versions VALUES ('version', '2')`); err != nil { |
| | | return err |
| | | } |
| | | |
| | | // 2. nodes 加列(逐列检查,幂等) |
| | | for _, col := range []struct{ name, def string }{ |
| | | {"aliases", "ALTER TABLE nodes ADD COLUMN aliases TEXT"}, |
| | | {"status", "ALTER TABLE nodes ADD COLUMN status TEXT"}, |
| | | {"size", "ALTER TABLE nodes ADD COLUMN size INTEGER"}, |
| | | {"mtime", "ALTER TABLE nodes ADD COLUMN mtime INTEGER"}, |
| | | {"content_hash", "ALTER TABLE nodes ADD COLUMN content_hash TEXT"}, |
| | | } { |
| | | if !s.nodeColumnExists(col.name) { |
| | | if _, err := s.db.Exec(col.def); err != nil { |
| | | return fmt.Errorf("加列 %s 失败: %w", col.name, err) |
| | | } |
| | | } |
| | | } |
| | | |
| | | // 3. edges 加 provenance 列 |
| | | if !s.edgeColumnExists("provenance") { |
| | | if _, err := s.db.Exec(`ALTER TABLE edges ADD COLUMN provenance TEXT`); err != nil { |
| | | return fmt.Errorf("edges 加 provenance 失败: %w", err) |
| | | } |
| | | } |
| | | |
| | | // 4. unresolved_links 表 |
| | | if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS unresolved_links ( |
| | | id INTEGER PRIMARY KEY AUTOINCREMENT, |
| | | from_node INTEGER NOT NULL, |
| | | link_text TEXT NOT NULL, |
| | | name_tail TEXT NOT NULL, |
| | | status TEXT NOT NULL DEFAULT 'pending', |
| | | created_at TEXT DEFAULT (datetime('now')), |
| | | last_attempt TEXT)`); err != nil { |
| | | return err |
| | | } |
| | | if _, err := s.db.Exec(`CREATE INDEX IF NOT EXISTS idx_unresolved_tail |
| | | ON unresolved_links(name_tail) WHERE status='failed'`); err != nil { |
| | | return err |
| | | } |
| | | |
| | | // 5. FTS 改 external-content:删旧表重建 + 触发器 + 重灌 |
| | | if _, err := s.db.Exec(`DROP TABLE IF EXISTS nodes_fts`); err != nil { |
| | | return err |
| | | } |
| | | if err := s.CreateFTS(); err != nil { |
| | | return err |
| | | } |
| | | return s.PopulateFTS() |
| | | } |
| | | |
| | | // nodeColumnExists / edgeColumnExists 幂等检查 |
| | | func (s *Store) nodeColumnExists(name string) bool { |
| | | var n int |
| | | s.db.QueryRow(`SELECT COUNT(*) FROM pragma_table_info('nodes') WHERE name=?`, name).Scan(&n) |
| | | return n > 0 |
| | | } |
| | | |
| | | func (s *Store) edgeColumnExists(name string) bool { |
| | | var n int |
| | | s.db.QueryRow(`SELECT COUNT(*) FROM pragma_table_info('edges') WHERE name=?`, name).Scan(&n) |
| | | return n > 0 |
| | | } |
| | | ``` |
| | | |
| | | - [ ] **Step 5: 改 Store 结构存 dbPath + Open 调 migrate** |
| | | |
| | | `internal/index/sqlite.go` 修改: |
| | | |
| | | ```go |
| | | // Store SQLite 存储层 |
| | | type Store struct { |
| | | db *sql.DB |
| | | dbPath string |
| | | } |
| | | |
| | | func Open(dbPath string) (*Store, error) { |
| | | dir := filepath.Dir(dbPath) |
| | | if err := os.MkdirAll(dir, 0755); err != nil { |
| | | return nil, fmt.Errorf("创建目录失败: %w", err) |
| | | } |
| | | db, err := sql.Open("sqlite3", dbPath+"?_journal_mode=WAL") |
| | | if err != nil { |
| | | return nil, fmt.Errorf("打开数据库失败: %w", err) |
| | | } |
| | | s := &Store{db: db, dbPath: dbPath} |
| | | if err := s.initTables(); err != nil { |
| | | db.Close() |
| | | return nil, err |
| | | } |
| | | if err := s.migrate(); err != nil { |
| | | db.Close() |
| | | return nil, err |
| | | } |
| | | return s, nil |
| | | } |
| | | ``` |
| | | |
| | | `initTables` 的 v1 建表语句保留(全新库先建 v1 再被 migrate 升到 v2,幂等)。 |
| | | |
| | | - [ ] **Step 6: 改 fts.go 为 external-content + 触发器** |
| | | |
| | | `internal/index/fts.go` 替换 CreateFTS/PopulateFTS: |
| | | |
| | | ```go |
| | | // CreateFTS 创建 FTS5 虚拟表(external-content 模式,由触发器增量维护) |
| | | func (s *Store) CreateFTS() error { |
| | | _, err := s.db.Exec(` |
| | | CREATE VIRTUAL TABLE IF NOT EXISTS nodes_fts USING fts5( |
| | | title, content, tags, aliases, |
| | | content='nodes', content_rowid='id' |
| | | )`) |
| | | if err != nil { |
| | | return err |
| | | } |
| | | // 触发器:nodes 的增删改同步维护 FTS |
| | | triggers := []string{ |
| | | `CREATE TRIGGER IF NOT EXISTS nodes_fts_ai AFTER INSERT ON nodes BEGIN |
| | | INSERT INTO nodes_fts(rowid, title, content, tags, aliases) |
| | | VALUES (new.id, new.title, new.content_fts, new.tags, new.aliases); |
| | | END`, |
| | | `CREATE TRIGGER IF NOT EXISTS nodes_fts_ad AFTER DELETE ON nodes BEGIN |
| | | INSERT INTO nodes_fts(nodes_fts, rowid, title, content, tags, aliases) |
| | | VALUES ('delete', old.id, old.title, old.content_fts, old.tags, old.aliases); |
| | | END`, |
| | | `CREATE TRIGGER IF NOT EXISTS nodes_fts_au AFTER UPDATE ON nodes BEGIN |
| | | INSERT INTO nodes_fts(nodes_fts, rowid, title, content, tags, aliases) |
| | | VALUES ('delete', old.id, old.title, old.content_fts, old.tags, old.aliases); |
| | | INSERT INTO nodes_fts(rowid, title, content, tags, aliases) |
| | | VALUES (new.id, new.title, new.content_fts, new.tags, new.aliases); |
| | | END`, |
| | | } |
| | | for _, t := range triggers { |
| | | if _, err := s.db.Exec(t); err != nil { |
| | | return err |
| | | } |
| | | } |
| | | return nil |
| | | } |
| | | |
| | | // PopulateFTS 全量重灌 FTS(external-content 模式专用语法) |
| | | func (s *Store) PopulateFTS() error { |
| | | _, err := s.db.Exec(`INSERT INTO nodes_fts(nodes_fts) VALUES('rebuild')`) |
| | | return err |
| | | } |
| | | ``` |
| | | |
| | | - [ ] **Step 7: 全量测试** |
| | | |
| | | Run: `CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./... 2>&1 | grep -v "no test files"` |
| | | Expected: 全部 PASS(含迁移测试 3 个 + 既有 FTS/search 测试) |
| | | |
| | | - [ ] **Step 8: Commit** |
| | | |
| | | ```bash |
| | | git add Makefile go.env internal/index/migrations.go internal/index/migrations_test.go internal/index/sqlite.go internal/index/fts.go |
| | | git commit -m "feat: schema v2 迁移(aliases/status/文件指纹/provenance/unresolved_links + FTS external-content)+ Makefile 固化 FTS5 构建标志" |
| | | ``` |
| | | |
| | | --- |
| | | |
| | | ### Task 2: 增量对账(reconcile) |
| | | |
| | | **Files:** |
| | | - Create: `internal/index/reconcile.go` |
| | | - Create: `internal/index/reconcile_test.go` |
| | | - Modify: `internal/vault/scanner.go`(新增 ScanVaultStat 只 stat 不解析) |
| | | - Modify: `internal/vault/parser.go`(Frontmatter 加 Status 字段) |
| | | - Modify: `internal/graph/model.go`(Node 加 Status/Aliases/Size/Mtime/ContentHash 字段) |
| | | - Modify: `internal/index/sqlite.go`(InsertNode 写新列 + UpsertNode + DeleteNode) |
| | | - Modify: `cmd/rebuild.go`(rebuildIndex 改走对账路径 + --force 全量) |
| | | - Modify: `cmd/index.go`(index build 默认增量,加 --force flag;index gc 保留) |
| | | - Delete: `internal/index/cache.go` 中 `NeedsRebuild` 函数(保留 GetGitCommit) |
| | | |
| | | **Interfaces:** |
| | | - Consumes: Task 1 的 v2 schema(nodes 新列) |
| | | - Produces: |
| | | - `func Reconcile(store *Store, vaultPath string) (*ReconcileResult, error)` |
| | | - `type ReconcileResult struct { Added, Modified, Deleted, Unchanged int }` |
| | | - `func QuickCheck(store *Store, vaultPath string) (bool, error)`(只 stat,返回"是否有差异") |
| | | - `func (s *Store) UpsertNode(n *graph.Node) error`(按 path INSERT OR REPLACE,触发器自动维护 FTS) |
| | | - `func (s *Store) DeleteNode(path string) error` |
| | | - `func (s *Store) GetFileStats() (map[string]FileStat, error)`;`type FileStat struct { Path string; Size int64; Mtime int64; ContentHash string }` |
| | | |
| | | - [ ] **Step 1: vault 层——Status 提取 + ScanVaultStat** |
| | | |
| | | `internal/vault/parser.go` Frontmatter 加字段 + ParseFile 赋值: |
| | | |
| | | ```go |
| | | type Frontmatter struct { |
| | | Title string `yaml:"title"` |
| | | Tags []string `yaml:"tags"` |
| | | Entities []string `yaml:"entities"` |
| | | Aliases []string `yaml:"aliases"` |
| | | Status string `yaml:"status"` |
| | | } |
| | | ``` |
| | | ParseFile 中解析成功后加:`meta.Status = fm.Status`;FileMeta 加 `Status string` 与 `Aliases []string` 字段(Aliases = fm.Aliases)。 |
| | | |
| | | `internal/vault/scanner.go` 加(不改 ScanVault): |
| | | |
| | | ```go |
| | | // FileStat 文件指纹(只 stat,不读内容) |
| | | type FileStat struct { |
| | | Path string // 相对路径 |
| | | Size int64 |
| | | Mtime int64 // Unix 秒 |
| | | } |
| | | |
| | | // ScanVaultStat 扫描 .md 文件指纹清单(跳过隐藏目录和待审阅目录,与 ScanVault 一致) |
| | | func ScanVaultStat(vaultPath string) ([]FileStat, error) { |
| | | var stats []FileStat |
| | | err := filepath.Walk(vaultPath, func(path string, info os.FileInfo, err error) error { |
| | | if err != nil { |
| | | return nil |
| | | } |
| | | if info.IsDir() { |
| | | if strings.HasPrefix(info.Name(), ".") && info.Name() != "." { |
| | | return filepath.SkipDir |
| | | } |
| | | if info.Name() == "待审阅" { |
| | | return filepath.SkipDir |
| | | } |
| | | return nil |
| | | } |
| | | if !strings.HasSuffix(path, ".md") { |
| | | return nil |
| | | } |
| | | relPath, _ := filepath.Rel(vaultPath, path) |
| | | stats = append(stats, FileStat{ |
| | | Path: relPath, |
| | | Size: info.Size(), |
| | | Mtime: info.ModTime().Unix(), |
| | | }) |
| | | return nil |
| | | }) |
| | | return stats, err |
| | | } |
| | | ``` |
| | | |
| | | - [ ] **Step 2: 写对账失败测试** |
| | | |
| | | `internal/index/reconcile_test.go`: |
| | | |
| | | ```go |
| | | package index |
| | | |
| | | import ( |
| | | "os" |
| | | "path/filepath" |
| | | "testing" |
| | | "time" |
| | | |
| | | "github.com/aisim/kb-cli/internal/graph" |
| | | "github.com/aisim/kb-cli/internal/vault" |
| | | ) |
| | | |
| | | // setupVault 造一个含 1 个 .md 的临时 vault |
| | | func setupVault(t *testing.T) string { |
| | | t.Helper() |
| | | dir := t.TempDir() |
| | | content := "---\ntitle: 测试文档\ntags: [t1]\n---\n正文提到 [[目标文档]] 和关键词补气\n" |
| | | if err := os.WriteFile(filepath.Join(dir, "FAQ", "001-测试文档.md"), []byte(content), 0644); err != nil { |
| | | // FAQ 目录需先建 |
| | | } |
| | | return dir |
| | | } |
| | | |
| | | func TestReconcileAdd(t *testing.T) { |
| | | dir := t.TempDir() |
| | | if err := os.MkdirAll(filepath.Join(dir, "FAQ"), 0755); err != nil { |
| | | t.Fatal(err) |
| | | } |
| | | content := "---\ntitle: 测试文档\ntags: [t1]\nstatus: 已解决\n---\n正文\n" |
| | | os.WriteFile(filepath.Join(dir, "FAQ", "001-测试文档.md"), []byte(content), 0644) |
| | | |
| | | store, _ := Open(filepath.Join(t.TempDir(), "kb.db")) |
| | | defer store.Close() |
| | | |
| | | res, err := Reconcile(store, dir) |
| | | if err != nil { |
| | | t.Fatal(err) |
| | | } |
| | | if res.Added != 1 || res.Unchanged != 0 { |
| | | t.Errorf("新增场景: %+v", res) |
| | | } |
| | | var title, status string |
| | | store.db.QueryRow(`SELECT title, status FROM nodes WHERE path='FAQ/001-测试文档.md'`).Scan(&title, &status) |
| | | if title != "测试文档" || status != "已解决" { |
| | | t.Errorf("节点字段错误: %s / %s", title, status) |
| | | } |
| | | } |
| | | |
| | | func TestReconcileModify(t *testing.T) { |
| | | dir := t.TempDir() |
| | | os.MkdirAll(filepath.Join(dir, "FAQ"), 0755) |
| | | p := filepath.Join(dir, "FAQ", "001-测试文档.md") |
| | | os.WriteFile(p, []byte("---\ntitle: 旧标题\n---\n旧内容\n"), 0644) |
| | | |
| | | store, _ := Open(filepath.Join(t.TempDir(), "kb.db")) |
| | | defer store.Close() |
| | | Reconcile(store, dir) |
| | | |
| | | // 修改内容(title 变新标题,内容加"补气") |
| | | time.Sleep(1100 * time.Millisecond) // mtime 秒级精度,确保 mtime 变化 |
| | | os.WriteFile(p, []byte("---\ntitle: 新标题\n---\n新内容补气\n"), 0644) |
| | | |
| | | res, err := Reconcile(store, dir) |
| | | if err != nil { |
| | | t.Fatal(err) |
| | | } |
| | | if res.Modified != 1 || res.Added != 0 { |
| | | t.Errorf("修改场景: %+v", res) |
| | | } |
| | | var title string |
| | | store.db.QueryRow(`SELECT title FROM nodes WHERE path='FAQ/001-测试文档.md'`).Scan(&title) |
| | | if title != "新标题" { |
| | | t.Errorf("标题未更新: %s", title) |
| | | } |
| | | } |
| | | |
| | | func TestReconcileDelete(t *testing.T) { |
| | | dir := t.TempDir() |
| | | os.MkdirAll(filepath.Join(dir, "FAQ"), 0755) |
| | | p := filepath.Join(dir, "FAQ", "001-测试文档.md") |
| | | os.WriteFile(p, []byte("---\ntitle: 测试文档\n---\n内容\n"), 0644) |
| | | |
| | | store, _ := Open(filepath.Join(t.TempDir(), "kb.db")) |
| | | defer store.Close() |
| | | Reconcile(store, dir) |
| | | os.Remove(p) |
| | | |
| | | res, err := Reconcile(store, dir) |
| | | if err != nil { |
| | | t.Fatal(err) |
| | | } |
| | | if res.Deleted != 1 { |
| | | t.Errorf("删除场景: %+v", res) |
| | | } |
| | | var n int |
| | | store.db.QueryRow(`SELECT COUNT(*) FROM nodes`).Scan(&n) |
| | | if n != 0 { |
| | | t.Errorf("节点未删净: %d", n) |
| | | } |
| | | } |
| | | |
| | | func TestReconcileUnchanged(t *testing.T) { |
| | | dir := t.TempDir() |
| | | os.MkdirAll(filepath.Join(dir, "FAQ"), 0755) |
| | | os.WriteFile(filepath.Join(dir, "FAQ", "001-测试文档.md"), |
| | | []byte("---\ntitle: 测试文档\n---\n内容\n"), 0644) |
| | | |
| | | store, _ := Open(filepath.Join(t.TempDir(), "kb.db")) |
| | | defer store.Close() |
| | | Reconcile(store, dir) |
| | | |
| | | res, err := Reconcile(store, dir) |
| | | if err != nil { |
| | | t.Fatal(err) |
| | | } |
| | | if res.Unchanged != 1 || res.Added != 0 || res.Modified != 0 { |
| | | t.Errorf("未变场景: %+v", res) |
| | | } |
| | | } |
| | | |
| | | func TestQuickCheck(t *testing.T) { |
| | | dir := t.TempDir() |
| | | os.MkdirAll(filepath.Join(dir, "FAQ"), 0755) |
| | | os.WriteFile(filepath.Join(dir, "FAQ", "001-测试文档.md"), |
| | | []byte("---\ntitle: 测试文档\n---\n内容\n"), 0644) |
| | | |
| | | store, _ := Open(filepath.Join(t.TempDir(), "kb.db")) |
| | | defer store.Close() |
| | | Reconcile(store, dir) |
| | | |
| | | dirty, err := QuickCheck(store, dir) |
| | | if err != nil { |
| | | t.Fatal(err) |
| | | } |
| | | if dirty { |
| | | t.Error("无差异时 QuickCheck 应 false") |
| | | } |
| | | } |
| | | ``` |
| | | |
| | | - [ ] **Step 3: 跑测试确认失败** |
| | | |
| | | Run: `CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./internal/index/ -run TestReconcile -v` |
| | | Expected: FAIL — `undefined: Reconcile` |
| | | |
| | | - [ ] **Step 4: Store 新方法(sqlite.go 追加)** |
| | | |
| | | ```go |
| | | // FileStat 索引中的文件指纹 |
| | | type FileStat struct { |
| | | Path string |
| | | Size int64 |
| | | Mtime int64 |
| | | ContentHash string |
| | | } |
| | | |
| | | // GetFileStats 返回所有已索引文件的指纹 |
| | | func (s *Store) GetFileStats() (map[string]FileStat, error) { |
| | | rows, err := s.db.Query(`SELECT path, size, mtime, content_hash FROM nodes`) |
| | | if err != nil { |
| | | return nil, err |
| | | } |
| | | defer rows.Close() |
| | | m := make(map[string]FileStat) |
| | | for rows.Next() { |
| | | var st FileStat |
| | | var hash sql.NullString |
| | | if err := rows.Scan(&st.Path, &st.Size, &st.Mtime, &hash); err != nil { |
| | | return nil, err |
| | | } |
| | | st.ContentHash = hash.String |
| | | m[st.Path] = st |
| | | } |
| | | return m, rows.Err() |
| | | } |
| | | |
| | | // UpsertNode 按 path 插入或更新节点(触发器自动维护 FTS) |
| | | func (s *Store) UpsertNode(n *graph.Node, size, mtime int64, contentHash string) error { |
| | | tagsJSON, _ := json.Marshal(n.Tags) |
| | | entitiesJSON, _ := json.Marshal(n.Entities) |
| | | wikilinksJSON, _ := json.Marshal(n.Wikilinks) |
| | | aliasesJSON, _ := json.Marshal(n.Aliases) |
| | | _, err := s.db.Exec(` |
| | | INSERT INTO nodes (path, title, section, tags, entities, wikilinks, aliases, status, |
| | | content_fts, size, mtime, content_hash) |
| | | VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) |
| | | ON CONFLICT(path) DO UPDATE SET |
| | | title = excluded.title, |
| | | section = excluded.section, |
| | | tags = excluded.tags, |
| | | entities = excluded.entities, |
| | | wikilinks = excluded.wikilinks, |
| | | aliases = excluded.aliases, |
| | | status = excluded.status, |
| | | content_fts = excluded.content_fts, |
| | | size = excluded.size, |
| | | mtime = excluded.mtime, |
| | | content_hash = excluded.content_hash, |
| | | updated_at = datetime('now')`, |
| | | n.Path, n.Title, n.Section, string(tagsJSON), string(entitiesJSON), |
| | | string(wikilinksJSON), string(aliasesJSON), n.Status, n.Content, size, mtime, contentHash) |
| | | return err |
| | | } |
| | | |
| | | // DeleteNode 按 path 删节点(级联删边),同时清该节点相关的 unresolved_links |
| | | func (s *Store) DeleteNode(path string) error { |
| | | tx, err := s.db.Begin() |
| | | if err != nil { |
| | | return err |
| | | } |
| | | var id int64 |
| | | if err := tx.QueryRow(`SELECT id FROM nodes WHERE path = ?`, path).Scan(&id); err != nil { |
| | | tx.Rollback() |
| | | return nil // 不存在,视为成功 |
| | | } |
| | | if _, err := tx.Exec(`DELETE FROM edges WHERE from_node = ? OR to_node = ?`, id, id); err != nil { |
| | | tx.Rollback() |
| | | return err |
| | | } |
| | | if _, err := tx.Exec(`DELETE FROM unresolved_links WHERE from_node = ?`, id); err != nil { |
| | | tx.Rollback() |
| | | return err |
| | | } |
| | | if _, err := tx.Exec(`DELETE FROM nodes WHERE id = ?`, id); err != nil { |
| | | tx.Rollback() |
| | | return err |
| | | } |
| | | return tx.Commit() |
| | | } |
| | | |
| | | // DeleteNodeEdges 只删某节点的边(保留节点行,用于"修改"场景重建边) |
| | | func (s *Store) DeleteNodeEdges(nodeID int64) error { |
| | | _, err := s.db.Exec(`DELETE FROM edges WHERE from_node = ? OR to_node = ?`, nodeID, nodeID) |
| | | return err |
| | | } |
| | | ``` |
| | | |
| | | 同时改 `InsertNode` 签名追加 `aliases/status/size/mtime/content_hash`(全量重建路径用),并同步 `graph.Node` 结构体加 `Aliases []string` 和 `Status string` 字段(Task 2 Step 1 已改 FileMeta,graph.Node 对齐)。 |
| | | |
| | | - [ ] **Step 5: reconcile.go 实现** |
| | | |
| | | ```go |
| | | package index |
| | | |
| | | import ( |
| | | "crypto/sha256" |
| | | "encoding/hex" |
| | | "fmt" |
| | | "os" |
| | | "path/filepath" |
| | | "strings" |
| | | |
| | | "github.com/aisim/kb-cli/internal/graph" |
| | | "github.com/aisim/kb-cli/internal/vault" |
| | | ) |
| | | |
| | | // ReconcileResult 对账结果统计 |
| | | type ReconcileResult struct { |
| | | Added int |
| | | Modified int |
| | | Deleted int |
| | | Unchanged int |
| | | } |
| | | |
| | | // Reconcile 增量对账:stat 比对 → 只对变更文件解析和写库 |
| | | func Reconcile(store *Store, vaultPath string) (*ReconcileResult, error) { |
| | | res := &ReconcileResult{} |
| | | |
| | | // 1. vault 侧指纹 |
| | | vaultStats, err := vault.ScanVaultStat(vaultPath) |
| | | if err != nil { |
| | | return nil, fmt.Errorf("扫描失败: %w", err) |
| | | } |
| | | dbStats, err := store.GetFileStats() |
| | | if err != nil { |
| | | return nil, fmt.Errorf("读取索引指纹失败: %w", err) |
| | | } |
| | | |
| | | // 2. 分类 |
| | | type change struct { |
| | | path string |
| | | stat vault.FileStat |
| | | known FileStat |
| | | } |
| | | var added, modified []change |
| | | for _, v := range vaultStats { |
| | | known, ok := dbStats[v.Path] |
| | | if !ok { |
| | | added = append(added, change{v.Path, v, FileStat{}}) |
| | | continue |
| | | } |
| | | if v.Size == known.Size && v.Mtime == known.Mtime { |
| | | res.Unchanged++ |
| | | continue |
| | | } |
| | | // size/mtime 变化 → sha256 二次确认 |
| | | hash, err := fileHash(filepath.Join(vaultPath, v.Path)) |
| | | if err != nil { |
| | | return nil, fmt.Errorf("哈希 %s 失败: %w", v.Path, err) |
| | | } |
| | | if known.ContentHash != "" && hash == known.ContentHash { |
| | | res.Unchanged++ // 内容没变(如 touch),只更新指纹 |
| | | store.db.Exec(`UPDATE nodes SET size=?, mtime=? WHERE path=?`, v.Size, v.Mtime, v.Path) |
| | | continue |
| | | } |
| | | modified = append(modified, change{v.Path, v, known}) |
| | | } |
| | | // 3. 删除 |
| | | for path := range dbStats { |
| | | found := false |
| | | for _, v := range vaultStats { |
| | | if v.Path == path { |
| | | found = true |
| | | break |
| | | } |
| | | } |
| | | if !found { |
| | | if err := store.DeleteNode(path); err != nil { |
| | | return nil, err |
| | | } |
| | | res.Deleted++ |
| | | } |
| | | } |
| | | |
| | | // 4. 新增 + 修改:解析 → 写节点 → 重建该节点边 |
| | | for _, c := range append(added, modified...) { |
| | | isNew := c.known.Path == "" |
| | | meta, err := vault.ParseFile(filepath.Join(vaultPath, c.path), c.path) |
| | | if err != nil { |
| | | return nil, fmt.Errorf("解析 %s 失败: %w", c.path, err) |
| | | } |
| | | hash, err := fileHash(filepath.Join(vaultPath, c.path)) |
| | | if err != nil { |
| | | return nil, err |
| | | } |
| | | if isNew { |
| | | res.Added++ |
| | | } else { |
| | | res.Modified++ |
| | | } |
| | | if err := applyNode(store, vaultPath, meta, c.stat.Size, c.stat.Mtime, hash); err != nil { |
| | | return nil, err |
| | | } |
| | | } |
| | | |
| | | // 5. 悬空链接重试(新增/修改节点可能让 failed 链接变可解析) |
| | | if res.Added+res.Modified > 0 { |
| | | store.RetryUnresolved() // 见 Task 4,此处先以空实现占位编译通过 |
| | | } |
| | | return res, nil |
| | | } |
| | | |
| | | func fileHash(path string) (string, error) { |
| | | data, err := os.ReadFile(path) |
| | | if err != nil { |
| | | return "", err |
| | | } |
| | | sum := sha256.Sum256(data) |
| | | return hex.EncodeToString(sum[:]), nil |
| | | } |
| | | |
| | | // applyNode 写节点 + 重建该节点的出边(tag/entity/wikilink)+ 悬空入表 |
| | | func applyNode(store *Store, vaultPath string, meta *vault.FileMeta, size, mtime int64, hash string) error { |
| | | n := &graph.Node{ |
| | | Path: meta.Path, Title: meta.Title, Section: meta.Section, |
| | | Tags: meta.Tags, Entities: meta.Entities, Wikilinks: meta.Wikilinks, |
| | | Aliases: meta.Aliases, Status: meta.Status, Content: meta.Content, |
| | | } |
| | | if err := store.UpsertNode(n, size, mtime, hash); err != nil { |
| | | return err |
| | | } |
| | | var nodeID int64 |
| | | if err := store.db.QueryRow(`SELECT id FROM nodes WHERE path=?`, meta.Path).Scan(&nodeID); err != nil { |
| | | return err |
| | | } |
| | | // 删旧边后重建出边 |
| | | if err := store.DeleteNodeEdges(nodeID); err != nil { |
| | | return err |
| | | } |
| | | return store.buildNodeEdges(nodeID, meta) |
| | | } |
| | | |
| | | // buildNodeEdges 为单个节点建出边;wikilink 解析失败入 unresolved_links |
| | | func (s *Store) buildNodeEdges(nodeID int64, meta *vault.FileMeta) error { |
| | | // tag / entity 边(虚拟节点 ID 沿用 1000000+ 规则,按 label 查现有行避免重复) |
| | | for _, tag := range meta.Tags { |
| | | if err := s.insertTagEntityEdge(nodeID, "tag:"+tag, "tag", tag); err != nil { |
| | | return err |
| | | } |
| | | } |
| | | for _, entity := range meta.Entities { |
| | | if err := s.insertTagEntityEdge(nodeID, "entity:"+entity, "entity", entity); err != nil { |
| | | return err |
| | | } |
| | | } |
| | | // wikilink 边 + 悬空 |
| | | for _, link := range meta.Wikilinks { |
| | | targetID, prov, ok := s.resolveWikilink(link) |
| | | if !ok { |
| | | tail := nameTail(link) |
| | | s.db.Exec(`INSERT INTO unresolved_links (from_node, link_text, name_tail, status) |
| | | VALUES (?, ?, ?, 'pending') ON CONFLICT DO NOTHING`, nodeID, link, tail) |
| | | continue |
| | | } |
| | | if err := s.InsertEdge(&graph.Edge{ |
| | | FromNode: nodeID, ToNode: targetID, |
| | | Relation: "wikilink", Label: link, Provenance: prov, |
| | | }); err != nil { |
| | | return err |
| | | } |
| | | } |
| | | return nil |
| | | } |
| | | |
| | | // insertTagEntityEdge tag/entity 边(虚拟节点按 label 复用 ID) |
| | | func (s *Store) insertTagEntityEdge(fromNode int64, key, relation, label string) error { |
| | | var virtualID int64 |
| | | err := s.db.QueryRow(`SELECT to_node FROM edges WHERE relation=? AND label=? LIMIT 1`, relation, label).Scan(&virtualID) |
| | | if err != nil { |
| | | // 新虚拟节点:分配 ID = 1000000 + 行号(稳定:按 label 排序后的行号) |
| | | var maxID int64 |
| | | s.db.QueryRow(`SELECT COALESCE(MAX(to_node), 1000000) FROM edges WHERE to_node >= 1000000 AND relation=?`, relation).Scan(&maxID) |
| | | virtualID = maxID + 1 |
| | | } |
| | | return s.InsertEdge(&graph.Edge{FromNode: fromNode, ToNode: virtualID, Relation: relation, Label: label}) |
| | | } |
| | | |
| | | // resolveWikilink 解析 wikilink 目标,返回 (nodeID, provenance, ok) |
| | | // provenance: exact = 标题或文件名精确匹配;fuzzy = 标题包含匹配 |
| | | func (s *Store) resolveWikilink(link string) (int64, string, bool) { |
| | | // 去锚点:[[标题|别名]] 取标题部分 |
| | | if idx := strings.Index(link, "|"); idx >= 0 { |
| | | link = link[:idx] |
| | | } |
| | | var id int64 |
| | | var title, path string |
| | | // 1. 标题精确 |
| | | err := s.db.QueryRow(`SELECT id, title, path FROM nodes WHERE title = ? LIMIT 1`, link).Scan(&id, &title, &path) |
| | | if err == nil { |
| | | return id, "exact", true |
| | | } |
| | | // 2. 文件名精确(去 .md 和编号前缀) |
| | | rows, err := s.db.Query(`SELECT id, title, path FROM nodes`) |
| | | if err != nil { |
| | | return 0, "", false |
| | | } |
| | | defer rows.Close() |
| | | var fuzzyID int64 |
| | | for rows.Next() { |
| | | var nid int64 |
| | | var nTitle, nPath string |
| | | if err := rows.Scan(&nid, &nTitle, &nPath); err != nil { |
| | | return 0, "", false |
| | | } |
| | | base := filepath.Base(nPath) |
| | | base = strings.TrimSuffix(base, ".md") |
| | | if dash := strings.Index(base, "-"); dash >= 0 { |
| | | base = base[dash+1:] |
| | | } |
| | | if base == link || nTitle == link { |
| | | return nid, "exact", true |
| | | } |
| | | if fuzzyID == 0 && strings.Contains(nTitle, link) { |
| | | fuzzyID = nid |
| | | } |
| | | } |
| | | if fuzzyID != 0 { |
| | | return fuzzyID, "fuzzy", true |
| | | } |
| | | return 0, "", false |
| | | } |
| | | |
| | | // nameTail 取 link 尾部用于重试匹配(去锚点修饰) |
| | | func nameTail(link string) string { |
| | | if idx := strings.Index(link, "|"); idx >= 0 { |
| | | link = link[:idx] |
| | | } |
| | | return link |
| | | } |
| | | ``` |
| | | |
| | | 注意:`resolveWikilink` 的全表扫描对 458 节点规模可接受;Task 4 的 RetryUnresolved 复用同一函数。`graph.Edge` 需加 `Provenance string` 字段,`InsertEdge` SQL 加 provenance 列。 |
| | | |
| | | - [ ] **Step 6: retryUnresolved 空实现占位(Task 4 补全)** |
| | | |
| | | reconcile.go 底部: |
| | | |
| | | ```go |
| | | // RetryUnresolved 重试解析悬空链接(Task 4 补全完整实现) |
| | | func (s *Store) RetryUnresolved() {} |
| | | ``` |
| | | |
| | | - [ ] **Step 7: 改 cmd 层** |
| | | |
| | | `cmd/rebuild.go`:`rebuildIndex` 保留为全量路径(改名 `rebuildFull`),新增: |
| | | |
| | | ```go |
| | | // syncIndex 增量对账路径(index build 默认、search pre-flight 用) |
| | | func syncIndex(store *index.Store) error { |
| | | res, err := index.Reconcile(store, vaultPath) |
| | | if err != nil { |
| | | return fmt.Errorf("增量同步失败: %w", err) |
| | | } |
| | | fmt.Fprintf(os.Stderr, "增量同步: 新增 %d / 修改 %d / 删除 %d / 未变 %d\n", |
| | | res.Added, res.Modified, res.Deleted, res.Unchanged) |
| | | if commit, err := index.GetGitCommit(vaultPath); err == nil { |
| | | store.SetMeta("git_commit", commit) |
| | | } |
| | | store.SetMeta("built_at", time.Now().Format(time.RFC3339)) |
| | | return nil |
| | | } |
| | | ``` |
| | | |
| | | `cmd/index.go`:`indexBuildCmd` 加 `--force` bool flag;`runIndexBuild`:force → rebuildFull,默认 → syncIndex。 |
| | | |
| | | `cmd/search.go`:pre-flight 从 `NeedsRebuild` 改为: |
| | | |
| | | ```go |
| | | dirty, err := index.QuickCheck(store, vaultPath) |
| | | if err != nil { |
| | | return fmt.Errorf("索引状态检查失败: %w", err) |
| | | } |
| | | if dirty { |
| | | fmt.Fprintln(os.Stderr, "索引有变更,正在增量同步...") |
| | | if err := syncIndex(store); err != nil { |
| | | return fmt.Errorf("增量同步失败: %w", err) |
| | | } |
| | | } |
| | | ``` |
| | | |
| | | `internal/index/cache.go`:删除 `NeedsRebuild` 函数(`GetGitCommit` 保留),文件重命名保持 cache.go 不改名(减少 diff)。 |
| | | |
| | | - [ ] **Step 8: QuickCheck 实现(reconcile.go)** |
| | | |
| | | ```go |
| | | // QuickCheck 只 stat 比对(不读内容不哈希),返回是否有差异 |
| | | func QuickCheck(store *Store, vaultPath string) (bool, error) { |
| | | vaultStats, err := vault.ScanVaultStat(vaultPath) |
| | | if err != nil { |
| | | return false, err |
| | | } |
| | | dbStats, err := store.GetFileStats() |
| | | if err != nil { |
| | | return false, err |
| | | } |
| | | if len(vaultStats) != len(dbStats) { |
| | | return true, nil |
| | | } |
| | | for _, v := range vaultStats { |
| | | known, ok := dbStats[v.Path] |
| | | if !ok || v.Size != known.Size || v.Mtime != known.Mtime { |
| | | return true, nil |
| | | } |
| | | } |
| | | return false, nil |
| | | } |
| | | ``` |
| | | |
| | | - [ ] **Step 9: 全量测试 + 编译** |
| | | |
| | | Run: `CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./... 2>&1 | grep -v "no test files" && CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go build ./...` |
| | | Expected: 全 PASS + 编译通过 |
| | | |
| | | - [ ] **Step 10: Commit** |
| | | |
| | | ```bash |
| | | git add internal/index/ internal/vault/ internal/graph/model.go cmd/ |
| | | git commit -m "feat: 增量对账同步(stat+sha256 比对,替代 commit hash 全量重建)" |
| | | ``` |
| | | |
| | | --- |
| | | |
| | | ### Task 3: 边可信度 + 悬空链接表接入全量重建 |
| | | |
| | | **Files:** |
| | | - Modify: `internal/graph/model.go`(Edge 加 Provenance) |
| | | - Modify: `internal/graph/builder.go`(BuildGraph 产出 provenance + Unresolved 列表) |
| | | - Create: `internal/graph/builder_test.go`(provenance 用例) |
| | | - Modify: `internal/index/sqlite.go`(InsertEdge 写 provenance;InsertUnresolved) |
| | | - Modify: `cmd/rebuild.go`(全量重建写入 unresolved) |
| | | |
| | | **Interfaces:** |
| | | - Consumes: Task 2 的 `resolveWikilink`(迁移为 graph 包共享函数 `graph.ResolveWikilink`,reconcile.go 改引用) |
| | | - Produces: |
| | | - `graph.Edge{Provenance string}`("exact"|"fuzzy"|"tag"|"entity") |
| | | - `graph.BuildGraph(files []*vault.FileMeta) (*Graph, []*UnresolvedLink)` |
| | | - `type UnresolvedLink struct { FromNode int64; LinkText string; NameTail string }`(FromNode 为 BuildGraph 内部 ID) |
| | | - `func (s *Store) InsertUnresolved(fromNode int64, linkText, nameTail string) error` |
| | | |
| | | - [ ] **Step 1: 失败测试——builder provenance** |
| | | |
| | | `internal/graph/builder_test.go`: |
| | | |
| | | ```go |
| | | package graph |
| | | |
| | | import ( |
| | | "testing" |
| | | |
| | | "github.com/aisim/kb-cli/internal/vault" |
| | | ) |
| | | |
| | | func TestBuildGraphProvenance(t *testing.T) { |
| | | files := []*vault.FileMeta{ |
| | | {Path: "FAQ/001-补气失败.md", Title: "补气失败", Wikilinks: []string{"红绿闪", "不存在的链接"}}, |
| | | {Path: "FAQ/002-红绿闪.md", Title: "红绿闪"}, |
| | | {Path: "知识/003-称重原理.md", Title: "称重原理详解"}, // "称重原理" 是 "称重原理详解" 的子串 → fuzzy |
| | | } |
| | | files[0].Wikilinks = append(files[0].Wikilinks, "称重原理") |
| | | |
| | | g, unresolved := BuildGraph(files) |
| | | |
| | | // exact: 标题精确匹配 |
| | | var exact, fuzzy int |
| | | for _, e := range g.Edges { |
| | | if e.Relation != "wikilink" { |
| | | continue |
| | | } |
| | | switch e.Provenance { |
| | | case "exact": |
| | | exact++ |
| | | case "fuzzy": |
| | | fuzzy++ |
| | | } |
| | | } |
| | | if exact != 1 || fuzzy != 1 { |
| | | t.Errorf("provenance 分布: exact=%d fuzzy=%d (want 1/1)", exact, fuzzy) |
| | | } |
| | | // 悬空链接 |
| | | if len(unresolved) != 1 || unresolved[0].LinkText != "不存在的链接" { |
| | | t.Errorf("unresolved: %+v", unresolved) |
| | | } |
| | | if unresolved[0].NameTail != "不存在的链接" { |
| | | t.Errorf("name_tail: %s", unresolved[0].NameTail) |
| | | } |
| | | } |
| | | ``` |
| | | |
| | | - [ ] **Step 2: 跑测试确认失败** |
| | | |
| | | Run: `CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./internal/graph/ -run TestBuildGraphProvenance -v` |
| | | Expected: FAIL — `Provenance 未定义 / BuildGraph 返回值不匹配` |
| | | |
| | | - [ ] **Step 3: 实现** |
| | | |
| | | `internal/graph/model.go`: |
| | | |
| | | ```go |
| | | // Edge 图边(实体关系) |
| | | type Edge struct { |
| | | FromNode int64 `json:"from_node"` |
| | | ToNode int64 `json:"to_node"` |
| | | Relation string `json:"relation"` // "tag" | "entity" | "wikilink" |
| | | Label string `json:"label"` |
| | | Provenance string `json:"provenance"` // "exact" | "fuzzy" | "tag" | "entity" |
| | | } |
| | | |
| | | // UnresolvedLink 悬空 wikilink(未匹配到目标节点) |
| | | type UnresolvedLink struct { |
| | | FromNode int64 // BuildGraph 内部节点 ID |
| | | LinkText string |
| | | NameTail string |
| | | } |
| | | ``` |
| | | |
| | | `internal/graph/builder.go`:`BuildGraph` 签名改 `func BuildGraph(files []*vault.FileMeta) (*Graph, []*UnresolvedLink)`;wikilink 匹配循环中: |
| | | |
| | | ```go |
| | | for _, n := range g.Nodes { |
| | | for _, link := range n.Wikilinks { |
| | | target, prov := matchWikilink(g.Nodes, link) |
| | | if target == nil { |
| | | unresolved = append(unresolved, &UnresolvedLink{ |
| | | FromNode: n.ID, LinkText: link, NameTail: nameTail(link), |
| | | }) |
| | | continue |
| | | } |
| | | g.Edges = append(g.Edges, &Edge{ |
| | | FromNode: n.ID, ToNode: target.ID, |
| | | Relation: "wikilink", Label: link, Provenance: prov, |
| | | }) |
| | | } |
| | | } |
| | | ``` |
| | | |
| | | `matchWikilink` 从现有 `matchesWikilink` 升级: |
| | | |
| | | ```go |
| | | // matchWikilink 返回 (目标节点, provenance)。exact = 标题/文件名精确;fuzzy = 标题包含 |
| | | func matchWikilink(nodes []*Node, link string) (*Node, string) { |
| | | // 去锚点 |
| | | if idx := strings.Index(link, "|"); idx >= 0 { |
| | | link = link[:idx] |
| | | } |
| | | var fuzzy *Node |
| | | for _, node := range nodes { |
| | | if node.Title == link { |
| | | return node, "exact" |
| | | } |
| | | base := filepath.Base(node.Path) |
| | | base = strings.TrimSuffix(base, ".md") |
| | | if dash := strings.Index(base, "-"); dash >= 0 { |
| | | base = base[dash+1:] |
| | | } |
| | | if base == link { |
| | | return node, "exact" |
| | | } |
| | | if fuzzy == nil && strings.Contains(node.Title, link) { |
| | | fuzzy = node |
| | | } |
| | | } |
| | | if fuzzy != nil { |
| | | return fuzzy, "fuzzy" |
| | | } |
| | | return nil, "" |
| | | } |
| | | |
| | | func nameTail(link string) string { |
| | | if idx := strings.Index(link, "|"); idx >= 0 { |
| | | link = link[:idx] |
| | | } |
| | | return link |
| | | } |
| | | ``` |
| | | |
| | | tag/entity 边追加 `Provenance: "tag"` / `"entity"`。 |
| | | |
| | | `internal/index/sqlite.go` InsertEdge 改: |
| | | |
| | | ```go |
| | | func (s *Store) InsertEdge(e *graph.Edge) error { |
| | | _, err := s.db.Exec(` |
| | | INSERT OR IGNORE INTO edges (from_node, to_node, relation, label, provenance) |
| | | VALUES (?, ?, ?, ?, ?)`, e.FromNode, e.ToNode, e.Relation, e.Label, e.Provenance) |
| | | return err |
| | | } |
| | | |
| | | // InsertUnresolved 悬空链接入表(幂等:同 from_node+link_text 不重复插) |
| | | func (s *Store) InsertUnresolved(fromNode int64, linkText, nameTail string) error { |
| | | _, err := s.db.Exec(` |
| | | INSERT INTO unresolved_links (from_node, link_text, name_tail, status) |
| | | SELECT ?, ?, ?, 'pending' |
| | | WHERE NOT EXISTS ( |
| | | SELECT 1 FROM unresolved_links WHERE from_node = ? AND link_text = ?)`, |
| | | fromNode, linkText, nameTail, fromNode, linkText) |
| | | return err |
| | | } |
| | | ``` |
| | | |
| | | `cmd/rebuild.go` 全量路径:`g, unresolved := graph.BuildGraph(files)`;节点写完后映射 ID 写 unresolved: |
| | | |
| | | ```go |
| | | for _, u := range unresolved { |
| | | fromID, ok := idMap[u.FromNode] |
| | | if !ok { |
| | | continue |
| | | } |
| | | if err := store.InsertUnresolved(fromID, u.LinkText, u.NameTail); err != nil { |
| | | return err |
| | | } |
| | | } |
| | | ``` |
| | | |
| | | `internal/index/reconcile.go`(Task 2 产物):`buildNodeEdges` 中 wikilink 解析改调 `graph.ResolveWikilink`——为便于共享,Task 2 的 `resolveWikilink` 逻辑保留在 Store 上(DB 查询版),graph 包的是纯内存版,两者规则一致即可,不强制合并(避免跨包依赖 SQLite)。 |
| | | |
| | | - [ ] **Step 4: 跑测试** |
| | | |
| | | Run: `CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./... 2>&1 | grep -v "no test files"` |
| | | Expected: 全 PASS |
| | | |
| | | - [ ] **Step 5: Commit** |
| | | |
| | | ```bash |
| | | git add internal/graph/ internal/index/sqlite.go cmd/rebuild.go |
| | | git commit -m "feat: wikilink 边可信度标注(exact/fuzzy)+ 悬空链接入 unresolved_links 表" |
| | | ``` |
| | | |
| | | --- |
| | | |
| | | ### Task 4: 悬空链接自动补全(RetryUnresolved) |
| | | |
| | | **Files:** |
| | | - Modify: `internal/index/reconcile.go`(RetryUnresolved 补全实现) |
| | | - Create: `internal/index/reconcile_test.go` 追加测试(或新文件 `unresolved_test.go`) |
| | | |
| | | **Interfaces:** |
| | | - Consumes: Task 3 的 `InsertUnresolved`、`resolveWikilink`(Store 版) |
| | | - Produces: `func (s *Store) RetryUnresolved() (resolved int, error error)`——每次对账后调用 |
| | | |
| | | - [ ] **Step 1: 失败测试** |
| | | |
| | | `internal/index/unresolved_test.go`: |
| | | |
| | | ```go |
| | | package index |
| | | |
| | | import ( |
| | | "os" |
| | | "path/filepath" |
| | | "testing" |
| | | ) |
| | | |
| | | func TestRetryUnresolved(t *testing.T) { |
| | | dir := t.TempDir() |
| | | os.MkdirAll(filepath.Join(dir, "FAQ"), 0755) |
| | | os.WriteFile(filepath.Join(dir, "FAQ", "001-旧文档.md"), |
| | | []byte("---\ntitle: 旧文档\n---\n提到 [[补气失败]]\n"), 0644) |
| | | |
| | | store, _ := Open(filepath.Join(t.TempDir(), "kb.db")) |
| | | defer store.Close() |
| | | |
| | | Reconcile(store, dir) |
| | | |
| | | // 验证悬空链接入表 |
| | | var n int |
| | | store.db.QueryRow(`SELECT COUNT(*) FROM unresolved_links WHERE link_text='补气失败'`).Scan(&n) |
| | | if n != 1 { |
| | | t.Fatalf("悬空链接未入表: %d", n) |
| | | } |
| | | |
| | | // 新文档入库:标题正好是 "补气失败" |
| | | os.WriteFile(filepath.Join(dir, "FAQ", "002-补气失败.md"), |
| | | []byte("---\ntitle: 补气失败\n---\n补气失败排查\n"), 0644) |
| | | |
| | | res, err := Reconcile(store, dir) |
| | | if err != nil { |
| | | t.Fatal(err) |
| | | } |
| | | if res.Added != 1 { |
| | | t.Fatalf("新增: %+v", res) |
| | | } |
| | | |
| | | // 悬空链接应被解析:行删除 + 边建立 |
| | | store.db.QueryRow(`SELECT COUNT(*) FROM unresolved_links WHERE link_text='补气失败'`).Scan(&n) |
| | | if n != 0 { |
| | | t.Errorf("悬空行未清除: %d", n) |
| | | } |
| | | store.db.QueryRow(`SELECT COUNT(*) FROM edges WHERE relation='wikilink' AND label='补气失败'`).Scan(&n) |
| | | if n != 1 { |
| | | t.Errorf("wikilink 边未建立: %d", n) |
| | | } |
| | | var prov string |
| | | store.db.QueryRow(`SELECT provenance FROM edges WHERE relation='wikilink' AND label='补气失败'`).Scan(&prov) |
| | | if prov != "exact" { |
| | | t.Errorf("补全边应为 exact: %s", prov) |
| | | } |
| | | } |
| | | ``` |
| | | |
| | | - [ ] **Step 2: 跑测试确认失败** |
| | | |
| | | Run: `CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./internal/index/ -run TestRetryUnresolved -v` |
| | | Expected: FAIL — RetryUnresolved 是空实现,悬空行不清除 |
| | | |
| | | - [ ] **Step 3: 实现 RetryUnresolved** |
| | | |
| | | reconcile.go 替换空实现: |
| | | |
| | | ```go |
| | | // RetryUnresolved 重试解析悬空链接:用当前全部节点的标题/文件名去匹配 unresolved_links 的 name_tail。 |
| | | // 命中则建边(provenance 按匹配严格度)、删行。返回成功解析条数。 |
| | | func (s *Store) RetryUnresolved() (int, error) { |
| | | rows, err := s.db.Query(`SELECT id, from_node, link_text, name_tail FROM unresolved_links`) |
| | | if err != nil { |
| | | return 0, err |
| | | } |
| | | type pending struct { |
| | | id int64 |
| | | fromNode int64 |
| | | linkText string |
| | | tail string |
| | | } |
| | | var pendings []pending |
| | | for rows.Next() { |
| | | var p pending |
| | | if err := rows.Scan(&p.id, &p.fromNode, &p.linkText, &p.tail); err != nil { |
| | | rows.Close() |
| | | return 0, err |
| | | } |
| | | pendings = append(pendings, p) |
| | | } |
| | | rows.Close() |
| | | if len(pendings) == 0 { |
| | | return 0, nil |
| | | } |
| | | |
| | | // 建匹配索引:标题/文件名(去编号) → nodeID,精确匹配优先 |
| | | type matchInfo struct { |
| | | id int64 |
| | | prov string |
| | | } |
| | | exactMap := make(map[string]matchInfo) |
| | | var fuzzyRows []struct { |
| | | id int64 |
| | | title string |
| | | } |
| | | nrows, err := s.db.Query(`SELECT id, title, path FROM nodes`) |
| | | if err != nil { |
| | | return 0, err |
| | | } |
| | | for nrows.Next() { |
| | | var id int64 |
| | | var title, path string |
| | | if err := nrows.Scan(&id, &title, &path); err != nil { |
| | | nrows.Close() |
| | | return 0, err |
| | | } |
| | | if _, ok := exactMap[title]; !ok { |
| | | exactMap[title] = matchInfo{id, "exact"} |
| | | } |
| | | base := filepath.Base(path) |
| | | base = strings.TrimSuffix(base, ".md") |
| | | if dash := strings.Index(base, "-"); dash >= 0 { |
| | | base = base[dash+1:] |
| | | } |
| | | if _, ok := exactMap[base]; !ok { |
| | | exactMap[base] = matchInfo{id, "exact"} |
| | | } |
| | | fuzzyRows = append(fuzzyRows, struct { |
| | | id int64 |
| | | title string |
| | | }{id, title}) |
| | | } |
| | | nrows.Close() |
| | | |
| | | resolved := 0 |
| | | for _, p := range pendings { |
| | | if info, ok := exactMap[p.tail]; ok { |
| | | // 建边 |
| | | if err := s.InsertEdge(&graph.Edge{ |
| | | FromNode: p.fromNode, ToNode: info.id, |
| | | Relation: "wikilink", Label: p.linkText, Provenance: info.prov, |
| | | }); err != nil { |
| | | return resolved, err |
| | | } |
| | | if _, err := s.db.Exec(`DELETE FROM unresolved_links WHERE id=?`, p.id); err != nil { |
| | | return resolved, err |
| | | } |
| | | resolved++ |
| | | continue |
| | | } |
| | | // fuzzy:标题包含 |
| | | for _, fr := range fuzzyRows { |
| | | if strings.Contains(fr.title, p.tail) { |
| | | if err := s.InsertEdge(&graph.Edge{ |
| | | FromNode: p.fromNode, ToNode: fr.id, |
| | | Relation: "wikilink", Label: p.linkText, Provenance: "fuzzy", |
| | | }); err != nil { |
| | | return resolved, err |
| | | } |
| | | s.db.Exec(`DELETE FROM unresolved_links WHERE id=?`, p.id) |
| | | resolved++ |
| | | break |
| | | } |
| | | } |
| | | } |
| | | return resolved, nil |
| | | } |
| | | ``` |
| | | |
| | | reconcile.go 需 import `github.com/aisim/kb-cli/internal/graph` 和 `strings`(已有)。 |
| | | |
| | | - [ ] **Step 4: 跑测试** |
| | | |
| | | Run: `CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./internal/index/ -v 2>&1 | tail -20` |
| | | Expected: 全 PASS(含 Task 2 的 5 个对账测试) |
| | | |
| | | - [ ] **Step 5: Commit** |
| | | |
| | | ```bash |
| | | git add internal/index/ |
| | | git commit -m "feat: 悬空双链自动补全(新文档入库后历史链接自动解析建边)" |
| | | ``` |
| | | |
| | | --- |
| | | |
| | | ### Task 5: RWR 图排序 + CJK 混合检索 + 双信号加权 |
| | | |
| | | **Files:** |
| | | - Create: `internal/graph/rwr.go` |
| | | - Create: `internal/graph/rwr_test.go` |
| | | - Create: `internal/index/graphload.go`(RWR 邻接加载) |
| | | - Modify: `internal/index/fts.go`(FTSSearch 拆 ASCII/CJK 双通道) |
| | | - Modify: `internal/search/engine.go`(双信号加权 + status 降权) |
| | | - Modify: `internal/search/scorer.go`(aliases 位置权重) |
| | | - Modify: `cmd/search.go`(读 config 的 text_weight) |
| | | - Modify: `internal/llm/client.go`(Config 加 Search/Explore 节) |
| | | |
| | | **Interfaces:** |
| | | - Consumes: Task 3 的 provenance 边 |
| | | - Produces: |
| | | - `func RWR(seedIDs []int64, adj map[int64][]adjEdge, alpha float64) map[int64]float64` |
| | | - `type adjEdge struct { To int64; Weight float64 }` |
| | | - `func (s *Store) LoadRWRGraph(seedIDs []int64) (map[int64][]adjEdge, error)`(只加载种子可达子图内的边;wikilink exact=1.0 / fuzzy=0.5 / entity=1.0 / tag=0.5) |
| | | - `func (s *Store) KeywordSearch(keywords []string, limit int) ([]FTSResult, error)`(ASCII 走 FTS MATCH,CJK 走 LIKE 合并) |
| | | - `func search.ContainsCJK(s string) bool` |
| | | |
| | | - [ ] **Step 1: RWR 失败测试** |
| | | |
| | | `internal/graph/rwr_test.go`: |
| | | |
| | | ```go |
| | | package graph |
| | | |
| | | import "testing" |
| | | |
| | | func TestRWRConvergenceAndSeed(t *testing.T) { |
| | | // 图: A-B-C 链 + D 孤立 |
| | | adj := map[int64][]adjEdge{ |
| | | 1: {{2, 1}}, 2: {{1, 1}, {3, 1}}, 3: {{2, 1}}, 4: {{}}, |
| | | } |
| | | r := RWR([]int64{1}, adj, 0.25) |
| | | if r[1] <= 0 { |
| | | t.Fatal("种子节点质量必须 > 0") |
| | | } |
| | | // 与种子连通的质量应高于孤立节点 |
| | | if r[4] >= r[3] { |
| | | t.Errorf("孤立节点质量不应高于连通节点: r4=%f r3=%f", r[4], r[3]) |
| | | } |
| | | // 归一化 [0,1] |
| | | for _, v := range r { |
| | | if v < 0 || v > 1 { |
| | | t.Errorf("质量未归一化: %f", v) |
| | | } |
| | | } |
| | | } |
| | | |
| | | func TestRWRWeightedByProvenance(t *testing.T) { |
| | | // A -exact-> B, A -fuzzy-> C:B 的质量应高于 C |
| | | adj := map[int64][]adjEdge{ |
| | | 1: {{2, 1.0}, {3, 0.5}}, |
| | | 2: {{1, 1.0}}, |
| | | 3: {{1, 0.5}}, |
| | | } |
| | | r := RWR([]int64{1}, adj, 0.25) |
| | | if r[2] <= r[3] { |
| | | t.Errorf("exact 边节点质量应高于 fuzzy: r2=%f r3=%f", r[2], r[3]) |
| | | } |
| | | } |
| | | ``` |
| | | |
| | | - [ ] **Step 2: 跑测试确认失败** |
| | | |
| | | Run: `CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./internal/graph/ -run TestRWR -v` |
| | | Expected: FAIL — `undefined: RWR` |
| | | |
| | | - [ ] **Step 3: 实现 rwr.go** |
| | | |
| | | ```go |
| | | package graph |
| | | |
| | | // adjEdge 带权邻接边 |
| | | type adjEdge struct { |
| | | To int64 |
| | | Weight float64 |
| | | } |
| | | |
| | | // RWR Random-Walk-with-Restart(个性化 PageRank): |
| | | // 从种子集合出发在无向带权图上做 power iteration,restart 概率 alpha 回到种子。 |
| | | // 返回归一化到 [0,1] 的游走质量。确定性算法,无随机数。 |
| | | func RWR(seedIDs []int64, adj map[int64][]adjEdge, alpha float64) map[int64]float64 { |
| | | if len(seedIDs) == 0 { |
| | | return map[int64]float64{} |
| | | } |
| | | // 收集参与节点:种子 + 邻接可达 |
| | | nodes := make(map[int64]bool) |
| | | for _, s := range seedIDs { |
| | | nodes[s] = true |
| | | } |
| | | for from := range adj { |
| | | nodes[from] = true |
| | | for _, e := range adj[from] { |
| | | nodes[e.To] = true |
| | | } |
| | | } |
| | | n := len(nodes) |
| | | if n == 0 { |
| | | return map[int64]float64{} |
| | | } |
| | | idx := make(map[int64]int, n) |
| | | ids := make([]int64, 0, n) |
| | | for id := range nodes { |
| | | idx[id] = len(ids) |
| | | ids = append(ids, id) |
| | | } |
| | | |
| | | // 重启向量:种子均匀 |
| | | r := make([]float64, n) |
| | | for _, s := range seedIDs { |
| | | if i, ok := idx[s]; ok { |
| | | r[i] = 1.0 / float64(len(seedIDs)) |
| | | } |
| | | } |
| | | |
| | | const maxIter = 50 |
| | | const eps = 1e-6 |
| | | newR := make([]float64, n) |
| | | for iter := 0; iter < maxIter; iter++ { |
| | | for i := range newR { |
| | | newR[i] = 0 |
| | | } |
| | | // 游走传播:out[i] = sum(w_ij) |
| | | for i, id := range ids { |
| | | var total float64 |
| | | for _, e := range adj[id] { |
| | | total += e.Weight |
| | | } |
| | | if total == 0 { |
| | | continue |
| | | } |
| | | for _, e := range adj[id] { |
| | | j, ok := idx[e.To] |
| | | if !ok { |
| | | continue |
| | | } |
| | | newR[j] += (1-alpha) * r[i] * e.Weight / total |
| | | } |
| | | } |
| | | // restart |
| | | for i := range newR { |
| | | newR[i] += alpha * r[i] |
| | | } |
| | | // 收敛判断 |
| | | diff := 0.0 |
| | | for i := range newR { |
| | | d := newR[i] - r[i] |
| | | if d < 0 { |
| | | d = -d |
| | | } |
| | | diff += d |
| | | } |
| | | r, newR = newR, r |
| | | if diff < eps { |
| | | break |
| | | } |
| | | } |
| | | // 归一化 [0,1] |
| | | maxV := 0.0 |
| | | for _, v := range r { |
| | | if v > maxV { |
| | | maxV = v |
| | | } |
| | | } |
| | | out := make(map[int64]float64, n) |
| | | if maxV == 0 { |
| | | for _, id := range ids { |
| | | out[id] = 0 |
| | | } |
| | | return out |
| | | } |
| | | for i, id := range ids { |
| | | out[id] = r[i] / maxV |
| | | } |
| | | return out |
| | | } |
| | | ``` |
| | | |
| | | - [ ] **Step 4: LoadRWRGraph(internal/index/graphload.go)** |
| | | |
| | | ```go |
| | | package index |
| | | |
| | | import ( |
| | | "fmt" |
| | | |
| | | "github.com/aisim/kb-cli/internal/graph" |
| | | ) |
| | | |
| | | // LoadRWRGraph 加载 RWR 邻接(无向、带权)。 |
| | | // 权重:wikilink exact=1.0 / fuzzy=0.5 / entity=1.0 / tag=0.5(tag 扇出大降权)。 |
| | | // 只加载与种子同连通域的边不可行(SQLite 无图查询),全量加载后 RWR 内部按种子收敛—— |
| | | // 458 节点 / ~2k 边规模下全量加载 <5ms,可接受。 |
| | | func (s *Store) LoadRWRGraph() (map[int64][]graph.adjEdge, error) { |
| | | rows, err := s.db.Query(` |
| | | SELECT from_node, to_node, relation, provenance FROM edges |
| | | WHERE relation IN ('wikilink', 'entity', 'tag')`) |
| | | if err != nil { |
| | | return nil, fmt.Errorf("加载边失败: %w", err) |
| | | } |
| | | defer rows.Close() |
| | | adj := make(map[int64][]graph.adjEdge) |
| | | add := func(from, to int64, w float64) { |
| | | adj[from] = append(adj[from], graph.adjEdge{To: to, Weight: w}) |
| | | adj[to] = append(adj[to], graph.adjEdge{To: from, Weight: w}) |
| | | } |
| | | for rows.Next() { |
| | | var from, to int64 |
| | | var relation, prov string |
| | | if err := rows.Scan(&from, &to, &relation, &prov); err != nil { |
| | | return nil, err |
| | | } |
| | | var w float64 |
| | | switch relation { |
| | | case "wikilink": |
| | | if prov == "exact" { |
| | | w = 1.0 |
| | | } else { |
| | | w = 0.5 |
| | | } |
| | | case "entity": |
| | | w = 1.0 |
| | | case "tag": |
| | | w = 0.5 |
| | | default: |
| | | continue |
| | | } |
| | | add(from, to, w) |
| | | } |
| | | return adj, rows.Err() |
| | | } |
| | | ``` |
| | | |
| | | - [ ] **Step 5: 跑测试** |
| | | |
| | | Run: `CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./internal/graph/ ./internal/index/ 2>&1 | tail -5` |
| | | Expected: PASS |
| | | |
| | | - [ ] **Step 6: CJK 混合检索(fts.go 加 KeywordSearch)** |
| | | |
| | | ```go |
| | | // containsCJK 是否含 CJK 统一表意文字 |
| | | func containsCJK(s string) bool { |
| | | for _, r := range s { |
| | | if r >= 0x4E00 && r <= 0x9FFF { |
| | | return true |
| | | } |
| | | } |
| | | return false |
| | | } |
| | | |
| | | // KeywordSearch 双通道关键词检索:ASCII 词走 FTS5 MATCH,CJK 词走 LIKE(title/aliases/content/tags)。 |
| | | // FTS5 unicode61 把连续中文当整串单 token,多字符 CJK 词 MATCH 匹配不到,必须走 LIKE。 |
| | | func (s *Store) KeywordSearch(keywords []string, limit int) ([]FTSResult, error) { |
| | | if len(keywords) == 0 { |
| | | return nil, nil |
| | | } |
| | | seen := make(map[int64]*FTSResult) |
| | | var order []int64 |
| | | |
| | | // 通道 1: FTS(ASCII 词) |
| | | var ascii []string |
| | | for _, kw := range keywords { |
| | | if !containsCJK(kw) { |
| | | ascii = append(ascii, kw) |
| | | } |
| | | } |
| | | for _, kw := range ascii { |
| | | rows, err := s.db.Query(` |
| | | SELECT n.id, n.path, n.title, n.section, fts.rank |
| | | FROM nodes_fts fts JOIN nodes n ON n.id = fts.rowid |
| | | WHERE nodes_fts MATCH ? ORDER BY fts.rank LIMIT ?`, kw, limit) |
| | | if err != nil { |
| | | continue // 语法异常词跳过 |
| | | } |
| | | for rows.Next() { |
| | | var r FTSResult |
| | | if err := rows.Scan(&r.ID, &r.Path, &r.Title, &r.Section, &r.Rank); err != nil { |
| | | continue |
| | | } |
| | | if _, ok := seen[r.ID]; !ok { |
| | | seen[r.ID] = &r |
| | | order = append(order, r.ID) |
| | | } |
| | | } |
| | | rows.Close() |
| | | } |
| | | |
| | | // 通道 2: LIKE(CJK 词) |
| | | for _, kw := range keywords { |
| | | if !containsCJK(kw) { |
| | | continue |
| | | } |
| | | pat := "%" + kw + "%" |
| | | rows, err := s.db.Query(` |
| | | SELECT id, path, title, section FROM nodes |
| | | WHERE title LIKE ? OR aliases LIKE ? OR content_fts LIKE ? OR tags LIKE ?`, |
| | | pat, pat, pat, pat) |
| | | if err != nil { |
| | | continue |
| | | } |
| | | for rows.Next() { |
| | | var r FTSResult |
| | | if err := rows.Scan(&r.ID, &r.Path, &r.Title, &r.Section); err != nil { |
| | | continue |
| | | } |
| | | r.Rank = 0 // LIKE 无 rank |
| | | if _, ok := seen[r.ID]; !ok { |
| | | seen[r.ID] = &r |
| | | order = append(order, r.ID) |
| | | } |
| | | } |
| | | rows.Close() |
| | | } |
| | | |
| | | results := make([]FTSResult, 0, len(order)) |
| | | for _, id := range order { |
| | | results = append(results, *seen[id]) |
| | | } |
| | | if limit > 0 && len(results) > limit { |
| | | results = results[:limit] |
| | | } |
| | | return results, nil |
| | | } |
| | | ``` |
| | | |
| | | - [ ] **Step 7: engine.go 双信号加权** |
| | | |
| | | 替换 `Search` 主体(保留函数签名,`SearchOptions` 加 `TextWeight float64`,缺省 0.5): |
| | | |
| | | ```go |
| | | func Search(store *index.Store, keywords []string, opts SearchOptions) ([]SearchResult, error) { |
| | | allKeywords := append(append(keywords, opts.Expanded...), opts.Symptom...) |
| | | |
| | | // 1. 双通道检索 |
| | | candidates, err := store.KeywordSearch(allKeywords, 100) |
| | | if err != nil { |
| | | return nil, err |
| | | } |
| | | if len(candidates) == 0 { |
| | | return nil, nil |
| | | } |
| | | |
| | | // 2. 文本分(沿用位置加权,aliases 按 title 档计权) |
| | | textScore := make(map[int64]int, len(candidates)) |
| | | for _, r := range candidates { |
| | | var score int |
| | | for _, kw := range keywords { |
| | | score += scoreResult(r, kw, ScoreNormal) |
| | | } |
| | | for _, kw := range opts.Expanded { |
| | | score += scoreResult(r, kw, ScoreExpanded) |
| | | } |
| | | for _, kw := range opts.Symptom { |
| | | score += scoreResult(r, kw, ScoreSymptom) |
| | | } |
| | | textScore[r.ID] = score |
| | | } |
| | | |
| | | // 3. RWR 图质量(种子 = 全部候选,上限 20) |
| | | seedIDs := make([]int64, 0, len(candidates)) |
| | | for _, r := range candidates { |
| | | if len(seedIDs) >= 20 { |
| | | break |
| | | } |
| | | seedIDs = append(seedIDs, r.ID) |
| | | } |
| | | rwrMass := map[int64]float64{} |
| | | if len(seedIDs) > 0 { |
| | | adj, err := store.LoadRWRGraph() |
| | | if err == nil { |
| | | rwrMass = graph.RWR(seedIDs, adj, 0.25) |
| | | } |
| | | } |
| | | |
| | | // 4. 双信号加权:finalScore = norm(textScore)*tw + rwrMass*(1-tw) |
| | | tw := opts.TextWeight |
| | | if tw <= 0 || tw > 1 { |
| | | tw = 0.5 |
| | | } |
| | | minT, maxT := 0, 0 |
| | | for _, s := range textScore { |
| | | if s < minT || maxT == 0 && s > maxT { |
| | | minT = s |
| | | } |
| | | if s > maxT { |
| | | maxT = s |
| | | } |
| | | } |
| | | statusFactor := make(map[int64]float64) |
| | | for _, r := range candidates { |
| | | if r.Section == "待审阅" || isDraftStatus(r.Status) { |
| | | statusFactor[r.ID] = 0.6 |
| | | } else { |
| | | statusFactor[r.ID] = 1.0 |
| | | } |
| | | } |
| | | |
| | | var results []SearchResult |
| | | for _, r := range candidates { |
| | | ts := float64(textScore[r.ID]) |
| | | if maxT > minT { |
| | | ts = (ts - float64(minT)) / float64(maxT-minT) |
| | | } else if maxT > 0 { |
| | | ts = 1 |
| | | } |
| | | final := ts*tw + rwrMass[r.ID]*(1-tw) |
| | | final *= statusFactor[r.ID] |
| | | results = append(results, SearchResult{ |
| | | ID: r.ID, Path: r.Path, Title: r.Title, Section: r.Section, |
| | | Score: int(final * 100), Status: r.Status, |
| | | }) |
| | | } |
| | | sort.Slice(results, func(i, j int) bool { |
| | | return results[i].Score > results[j].Score |
| | | }) |
| | | if opts.TopN > 0 && len(results) > opts.TopN { |
| | | results = results[:opts.TopN] |
| | | } |
| | | // WithContent / WithLinks 逻辑沿用现有代码 |
| | | if opts.WithContent { |
| | | for i := range results { |
| | | content, _, _, err := store.GetNodeContent(results[i].ID) |
| | | if err == nil { |
| | | results[i].Content = content |
| | | } |
| | | } |
| | | } |
| | | if opts.WithLinks { |
| | | for i := range results { |
| | | links, err := store.GetNodeLinks(results[i].ID) |
| | | if err == nil { |
| | | results[i].Links = links |
| | | } |
| | | } |
| | | } |
| | | return results, nil |
| | | } |
| | | |
| | | // isDraftStatus 草稿态降权判断 |
| | | func isDraftStatus(status string) bool { |
| | | switch status { |
| | | case "草稿", "待确认", "跟进中": |
| | | return true |
| | | } |
| | | return false |
| | | } |
| | | ``` |
| | | |
| | | 配套修改: |
| | | - `FTSResult` 加 `Status string` 字段;`KeywordSearch` 两通道 SELECT 均加 `n.status` 列并 Scan |
| | | - `SearchResult` 加 `Status string \`json:"status,omitempty"\`` |
| | | - `SearchOptions` 加 `TextWeight float64` |
| | | - `scoreResult` 加 aliases 检查:`var aliasesJSON string` 从 store 取(或 FTSResult 带 Aliases []string 解析),命中按 "title" 档计权: |
| | | |
| | | ```go |
| | | // scoreResult 追加(Aliases 字段已在 FTSResult 上): |
| | | if len(r.Aliases) > 0 { |
| | | for _, a := range r.Aliases { |
| | | if strings.Contains(strings.ToLower(a), kw) { |
| | | score += CalcScore(keyword, "title", scoreType) |
| | | break |
| | | } |
| | | } |
| | | } |
| | | ``` |
| | | (`FTSResult` 加 `Aliases []string`;KeywordSearch SELECT 加 `n.aliases` 并 json.Unmarshal) |
| | | |
| | | - `cmd/search.go`:`opts.TextWeight` 从 config 读(`cfg.Search.TextWeight`,缺省 0.5) |
| | | - `internal/llm/client.go` Config 加: |
| | | |
| | | ```go |
| | | Search struct { |
| | | TextWeight float64 `yaml:"text_weight"` |
| | | } `yaml:"search"` |
| | | Explore struct { |
| | | DefaultBudget int `yaml:"default_budget"` |
| | | HardBudget int `yaml:"hard_budget"` |
| | | TopN int `yaml:"top_n"` |
| | | } `yaml:"explore"` |
| | | ``` |
| | | |
| | | - [ ] **Step 8: engine_test.go 适配 + 新测试** |
| | | |
| | | 现有 `TestSearch` 保持通过(FTS 通道不变)。追加: |
| | | |
| | | ```go |
| | | func TestCJKLikeChannel(t *testing.T) { |
| | | // 库中含 "电子秤补气失败" 的文档,搜 "补气"(CJK)应命中 |
| | | // 构造:store.UpsertNode 写入 content 含该词的节点 |
| | | ... |
| | | } |
| | | ``` |
| | | |
| | | 测试构造方式:`Open` 临时库 → `UpsertNode` 写 3 个节点(1 个 content 含"补气",1 个 title 含"补气",1 个不含)→ `Search(store, []string{"补气"}, opts)` → 断言含"补气"的两个在结果中且 title 命中者排前。 |
| | | |
| | | - [ ] **Step 9: 全量测试** |
| | | |
| | | Run: `CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./... 2>&1 | grep -v "no test files"` |
| | | Expected: 全 PASS |
| | | |
| | | - [ ] **Step 10: Commit** |
| | | |
| | | ```bash |
| | | git add internal/graph/rwr.go internal/graph/rwr_test.go internal/index/graphload.go internal/index/fts.go internal/search/ internal/llm/client.go cmd/search.go |
| | | git commit -m "feat: RWR 图结构排序 + CJK LIKE 混合检索 + 双信号加权 + aliases/status 参与排序" |
| | | ``` |
| | | |
| | | --- |
| | | |
| | | ### Task 6: explore 命令 |
| | | |
| | | **Files:** |
| | | - Create: `cmd/explore.go` |
| | | - Create: `internal/search/explore.go` |
| | | - Create: `internal/search/explore_test.go` |
| | | - Modify: `cmd/root.go`(注册 exploreCmd) |
| | | |
| | | **Interfaces:** |
| | | - Consumes: Task 5 的 `Search`、`store.GetNodeContent`、`store.GetNodeLinks` |
| | | - Produces: |
| | | - `func Explore(store *index.Store, query string, opts ExploreOptions) (*ExploreResult, error)` |
| | | - `type ExploreOptions struct { Budget, TopN int }` |
| | | - `type ExploreResult struct { Docs []ExploredDoc; Related map[string][]string; UnresolvedLinks []string }` |
| | | - `type ExploredDoc struct { Path, Title, Section string; Score int; Body string }` |
| | | |
| | | - [ ] **Step 1: 失败测试** |
| | | |
| | | `internal/search/explore_test.go`: |
| | | |
| | | ```go |
| | | package search |
| | | |
| | | import ( |
| | | "testing" |
| | | ) |
| | | |
| | | // TestExploreParagraphExtraction 段落截取:只输出命中关键词的段落,整段不截半句 |
| | | func TestExploreParagraphExtraction(t *testing.T) { |
| | | content := "# 标题\n\n第一段讲称重。\n\n## 补气流程\n\n补气失败时先检查阀门。\n\n## 其他\n\n无关内容。\n" |
| | | got := extractRelevantParagraphs(content, []string{"补气"}) |
| | | want := "## 补气流程\n\n补气失败时先检查阀门。\n" |
| | | if got != want { |
| | | t.Errorf("段落截取:\n got=%q\nwant=%q", got, want) |
| | | } |
| | | } |
| | | |
| | | // TestExploreWholeDocWhenSmall 文档短于预算时整篇输出 |
| | | func TestExploreWholeDocWhenSmall(t *testing.T) { |
| | | got := extractRelevantParagraphs("短文档\n", []string{"不存在"}, 10000) |
| | | if got != "短文档\n" { |
| | | t.Errorf("应整篇输出: %q", got) |
| | | } |
| | | } |
| | | |
| | | // TestExploreBudget 预算分配:按分数降序,超预算文档截断到段落边界 |
| | | func TestExploreBudget(t *testing.T) { |
| | | // 用 extractRelevantParagraphs 的预算版验证:预算 20 字节,命中段落 30 字节 → 输出空(宁缺毋滥,不截半段) |
| | | got := extractWithBudget("## 段落\n\n这是一段超过预算的内容啊\n", []string{"段落"}, 20) |
| | | if got != "" { |
| | | t.Errorf("超预算段落应跳过: %q", got) |
| | | } |
| | | } |
| | | ``` |
| | | |
| | | - [ ] **Step 2: 跑测试确认失败** |
| | | |
| | | Run: `CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./internal/search/ -run TestExplore -v` |
| | | Expected: FAIL — `undefined: extractRelevantParagraphs` |
| | | |
| | | - [ ] **Step 3: 实现 explore.go** |
| | | |
| | | ```go |
| | | package search |
| | | |
| | | import ( |
| | | "regexp" |
| | | "strings" |
| | | |
| | | "github.com/aisim/kb-cli/internal/index" |
| | | ) |
| | | |
| | | var headingRe = regexp.MustCompile(`^#{1,4} .+$`) |
| | | |
| | | // splitParagraphs 按 1-4 级标题切段。无标题的文档整体为一段。 |
| | | func splitParagraphs(content string) []string { |
| | | lines := strings.Split(content, "\n") |
| | | var paras []string |
| | | var cur []string |
| | | flush := func() { |
| | | if len(cur) > 0 { |
| | | paras = append(paras, strings.TrimRight(strings.Join(cur, "\n"), "\n")+"\n") |
| | | cur = nil |
| | | } |
| | | } |
| | | for _, l := range lines { |
| | | if headingRe.MatchString(l) { |
| | | flush() |
| | | } |
| | | cur = append(cur, l) |
| | | } |
| | | flush() |
| | | return paras |
| | | } |
| | | |
| | | // extractRelevantParagraphs 返回命中关键词的段落(整段不截半句)。 |
| | | // 文档总长 <= budget 时整篇输出;无命中段落时输出空串。 |
| | | func extractRelevantParagraphs(content string, keywords []string, budget int) string { |
| | | if budget <= 0 || len(content) <= budget { |
| | | return content |
| | | } |
| | | var out []string |
| | | for _, p := range splitParagraphs(content) { |
| | | for _, kw := range keywords { |
| | | if strings.Contains(p, kw) { |
| | | out = append(out, p) |
| | | break |
| | | } |
| | | } |
| | | } |
| | | // 预算约束:累计超预算的段落丢弃(不截半段) |
| | | var total int |
| | | kept := []string{} |
| | | for _, p := range out { |
| | | if total+len(p) > budget { |
| | | break |
| | | } |
| | | total += len(p) |
| | | kept = append(kept, p) |
| | | } |
| | | return strings.Join(kept, "") |
| | | } |
| | | |
| | | // extractWithBudget extractRelevantParagraphs 的预算版(budget<=0 视为无预算) |
| | | func extractWithBudget(content string, keywords []string, budget int) string { |
| | | if budget <= 0 { |
| | | budget = 100000 |
| | | } |
| | | return extractRelevantParagraphs(content, keywords, budget) |
| | | } |
| | | |
| | | // ExploreOptions explore 参数 |
| | | type ExploreOptions struct { |
| | | Budget int // 字节预算(0 = 用配置默认) |
| | | TopN int // 0 = 用配置默认 |
| | | } |
| | | |
| | | // ExploredDoc 入选文档及其输出正文 |
| | | type ExploredDoc struct { |
| | | Path string `json:"path"` |
| | | Title string `json:"title"` |
| | | Section string `json:"section"` |
| | | Score int `json:"score"` |
| | | Body string `json:"body"` |
| | | } |
| | | |
| | | // ExploreResult explore 结果 |
| | | type ExploreResult struct { |
| | | Docs []ExploredDoc `json:"docs"` |
| | | Related map[string][]string `json:"related"` |
| | | UnresolvedLinks []string `json:"unresolved_links"` |
| | | } |
| | | |
| | | // Explore 一次调用返回相关文档原文 + 关联清单 + 悬空链接 |
| | | func Explore(store *index.Store, query string, cfg ExploreOptions) (*ExploreResult, error) { |
| | | keywords := strings.Fields(query) |
| | | if len(keywords) == 0 { |
| | | return nil, nil |
| | | } |
| | | budget := cfg.Budget |
| | | if budget <= 0 { |
| | | budget = 16000 |
| | | } |
| | | if budget > 32000 { |
| | | budget = 32000 |
| | | } |
| | | topN := cfg.TopN |
| | | if topN <= 0 { |
| | | topN = 5 |
| | | } |
| | | |
| | | opts := SearchOptions{TopN: topN} |
| | | results, err := Search(store, keywords, opts) |
| | | if err != nil { |
| | | return nil, err |
| | | } |
| | | if len(results) == 0 { |
| | | return &ExploreResult{Related: map[string][]string{}}, nil |
| | | } |
| | | |
| | | // 按分数降序分配预算:每文档至少 800 字节 |
| | | res := &ExploreResult{Related: map[string][]string{}} |
| | | perDoc := budget / len(results) |
| | | if perDoc < 800 { |
| | | perDoc = 800 |
| | | } |
| | | for i, r := range results { |
| | | content, _, _, err := store.GetNodeContent(r.ID) |
| | | if err != nil || content == "" { |
| | | continue |
| | | } |
| | | body := extractRelevantParagraphs(content, keywords, perDoc) |
| | | if body == "" { |
| | | // 无命中段落但文档入选 → 整篇(若放得下),否则跳过 |
| | | if len(content) <= perDoc { |
| | | body = content |
| | | } else { |
| | | continue |
| | | } |
| | | } |
| | | res.Docs = append(res.Docs, ExploredDoc{ |
| | | Path: r.Path, Title: r.Title, Section: r.Section, |
| | | Score: r.Score, Body: body, |
| | | }) |
| | | // 关联清单 |
| | | if links, err := store.GetNodeLinks(r.ID); err == nil { |
| | | res.Related[r.Path] = links |
| | | } |
| | | _ = i |
| | | } |
| | | |
| | | // 悬空链接提示:入选文档的 wikilinks 中未解析的 |
| | | for _, d := range res.Docs { |
| | | links, err := store.GetUnresolvedLinks(d.Path) |
| | | if err == nil { |
| | | res.UnresolvedLinks = append(res.UnresolvedLinks, links...) |
| | | } |
| | | } |
| | | return res, nil |
| | | } |
| | | ``` |
| | | |
| | | Store 需补方法(sqlite.go): |
| | | |
| | | ```go |
| | | // GetUnresolvedLinks 某文档的悬空链接文本列表 |
| | | func (s *Store) GetUnresolvedLinks(path string) ([]string, error) { |
| | | rows, err := s.db.Query(` |
| | | SELECT u.link_text FROM unresolved_links u |
| | | JOIN nodes n ON n.id = u.from_node WHERE n.path = ?`) |
| | | if err != nil { |
| | | return nil, err |
| | | } |
| | | defer rows.Close() |
| | | var links []string |
| | | for rows.Next() { |
| | | var l string |
| | | rows.Scan(&l) |
| | | links = append(links, l) |
| | | } |
| | | return links, rows.Err() |
| | | } |
| | | ``` |
| | | |
| | | - [ ] **Step 4: cmd/explore.go** |
| | | |
| | | ```go |
| | | package cmd |
| | | |
| | | import ( |
| | | "fmt" |
| | | "os" |
| | | |
| | | "github.com/aisim/kb-cli/internal/index" |
| | | "github.com/aisim/kb-cli/internal/search" |
| | | "github.com/spf13/cobra" |
| | | ) |
| | | |
| | | var ( |
| | | exploreBudget int |
| | | exploreTopN int |
| | | exploreJSON bool |
| | | ) |
| | | |
| | | var exploreCmd = &cobra.Command{ |
| | | Use: "explore <问题>", |
| | | Short: "一次调用获取相关文档原文 + 关联清单(供 agent 使用)", |
| | | Long: `# explore - 精准上下文 |
| | | kb-cli explore <问题> [--budget 字节] [--top N] [--json] # 按字节预算返回相关文档原文、关联文档、悬空链接`, |
| | | Args: cobra.MinimumNArgs(1), |
| | | RunE: runExplore, |
| | | } |
| | | |
| | | func init() { |
| | | rootCmd.AddCommand(exploreCmd) |
| | | exploreCmd.Flags().IntVar(&exploreBudget, "budget", 0, "字节预算(0=配置默认 16000)") |
| | | exploreCmd.Flags().IntVar(&exploreTopN, "top", 0, "文档数(0=配置默认 5)") |
| | | exploreCmd.Flags().BoolVar(&exploreJSON, "json", false, "JSON 输出") |
| | | exploreCmd.SetUsageTemplate(exploreCmd.Long) |
| | | } |
| | | |
| | | func runExplore(cmd *cobra.Command, args []string) error { |
| | | store, err := index.Open(dbPath) |
| | | if err != nil { |
| | | return fmt.Errorf("打开索引失败: %w", err) |
| | | } |
| | | defer store.Close() |
| | | |
| | | dirty, err := index.QuickCheck(store, vaultPath) |
| | | if err != nil { |
| | | return fmt.Errorf("索引状态检查失败: %w", err) |
| | | } |
| | | if dirty { |
| | | fmt.Fprintln(os.Stderr, "索引有变更,正在增量同步...") |
| | | if err := syncIndex(store); err != nil { |
| | | return fmt.Errorf("增量同步失败: %w", err) |
| | | } |
| | | } |
| | | |
| | | query := strings.Join(args, " ") |
| | | res, err := search.Explore(store, query, search.ExploreOptions{Budget: exploreBudget, TopN: exploreTopN}) |
| | | if err != nil { |
| | | return fmt.Errorf("explore 失败: %w", err) |
| | | } |
| | | if len(res.Docs) == 0 { |
| | | fmt.Println("未找到相关文档") |
| | | return nil |
| | | } |
| | | |
| | | if exploreJSON { |
| | | b, _ := json.MarshalIndent(res, "", " ") |
| | | fmt.Println(string(b)) |
| | | return nil |
| | | } |
| | | for _, d := range res.Docs { |
| | | fmt.Printf("## %s(%s,score %d)\n", d.Title, d.Path, d.Score) |
| | | fmt.Print(d.Body) |
| | | fmt.Println() |
| | | if rel, ok := res.Related[d.Path]; ok && len(rel) > 0 { |
| | | fmt.Printf("关联: %s\n", strings.Join(rel, ", ")) |
| | | } |
| | | fmt.Println() |
| | | } |
| | | if len(res.UnresolvedLinks) > 0 { |
| | | fmt.Printf("⚠️ 悬空链接: %s\n", strings.Join(res.UnresolvedLinks, ", ")) |
| | | } |
| | | fmt.Println("以上为文档原文直出,agent 无需再读文件") |
| | | return nil |
| | | } |
| | | ``` |
| | | |
| | | import 需加 `encoding/json` 和 `strings`。 |
| | | |
| | | - [ ] **Step 5: 跑测试 + 编译** |
| | | |
| | | Run: `CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./... 2>&1 | grep -v "no test files" && CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go build -o kb-cli .` |
| | | Expected: 全 PASS + 二进制产出 |
| | | |
| | | - [ ] **Step 6: 手动冒烟** |
| | | |
| | | ```bash |
| | | CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go build -o kb-cli . |
| | | ./kb-cli index build --vault ~/rag-lpg-obsidian |
| | | ./kb-cli explore "电子秤补气失败" --top 3 |
| | | ``` |
| | | Expected: 输出含补气相关文档原文段落 + 关联清单 + "原文直出"尾注 |
| | | |
| | | - [ ] **Step 7: Commit** |
| | | |
| | | ```bash |
| | | git add cmd/explore.go internal/search/explore.go internal/search/explore_test.go internal/index/sqlite.go |
| | | git commit -m "feat: explore 命令(字节预算 + 段落级原文直出 + 关联清单 + 悬空链接提示)" |
| | | ``` |
| | | |
| | | --- |
| | | |
| | | ### Task 7: 端到端验证 + README |
| | | |
| | | **Files:** |
| | | - Modify: `README.md`(命令文档更新) |
| | | - Create: `scripts/e2e-verify.sh` |
| | | |
| | | **Interfaces:** |
| | | - Consumes: Task 1-6 全部产物 |
| | | |
| | | - [ ] **Step 1: e2e 验证脚本** |
| | | |
| | | `scripts/e2e-verify.sh`: |
| | | |
| | | ```bash |
| | | #!/bin/bash |
| | | # 端到端验证(对标 spec 测试计划第 2 节) |
| | | set -e |
| | | export CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" |
| | | export CGO_LDFLAGS="-lm" |
| | | VAULT="${1:-$HOME/rag-lpg-obsidian}" |
| | | cd "$(dirname "$0")/.." |
| | | go build -o /tmp/kb-cli-e2e . |
| | | |
| | | echo "=== 1. 全量重建基准 ===" |
| | | /tmp/kb-cli-e2e index build --vault "$VAULT" --force 2>&1 | tail -1 |
| | | |
| | | echo "=== 2. 增量同步耗时 ===" |
| | | time /tmp/kb-cli-e2e index build --vault "$VAULT" 2>&1 | tail -1 |
| | | |
| | | echo "=== 3. 未 commit 编辑可见性 ===" |
| | | TESTFILE="$VAULT/笔记/e2e-test-$(date +%s).md" |
| | | printf -- "---\ntitle: e2e测试\ntags: [补气]\n---\ne2e 补气测试内容\n" > "$TESTFILE" |
| | | /tmp/kb-cli-e2e search e2e --vault "$VAULT" --top 3 2>&1 | grep -q "e2e测试" && echo "PASS: 未commit编辑可见" || echo "FAIL: 未commit编辑不可见" |
| | | rm -f "$TESTFILE" |
| | | /tmp/kb-cli-e2e index build --vault "$VAULT" 2>/dev/null |
| | | |
| | | echo "=== 4. CJK 召回质量(补气应命中多条)===" |
| | | N=$(/tmp/kb-cli-e2e search 补气 --vault "$VAULT" --top 10 2>/dev/null | grep -c "^文档/\|^FAQ/\|^知识/" || true) |
| | | echo "补气 命中 $N 条 (期望 >= 5)" |
| | | |
| | | echo "=== 5. explore 预算内 ===" |
| | | BYTES=$(/tmp/kb-cli-e2e explore "电子秤补气失败" --vault "$VAULT" --top 3 2>/dev/null | wc -c) |
| | | echo "explore 输出 ${BYTES} 字节 (上限 32000+尾注)" |
| | | |
| | | echo "=== 6. 悬空补全 ===" |
| | | /tmp/kb-cli-e2e graph stats --vault "$VAULT" 2>&1 | head -5 |
| | | ``` |
| | | |
| | | - [ ] **Step 2: 跑 e2e** |
| | | |
| | | Run: `bash scripts/e2e-verify.sh` |
| | | Expected: 第 3 项 PASS;第 4 项 N>=5(改造前实测为 1);第 5 项 BYTES <= 32200 |
| | | |
| | | - [ ] **Step 3: README 更新** |
| | | |
| | | README.md 命令章节更新: |
| | | - `index build` 说明改"默认增量对账同步,`--force` 全量重建" |
| | | - `search` 说明加"ASCII 走 FTS5、CJK 走 LIKE 双通道;排序 = 文本分 + RWR 图质量双信号" |
| | | - 新增 `explore` 命令说明 |
| | | - 新增"构建要求"章节:`CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go build`(或 make build) |
| | | - `graph query/related` 输出含 provenance 标注说明 |
| | | |
| | | - [ ] **Step 4: 最终全量测试 + 提交推送** |
| | | |
| | | ```bash |
| | | CGO_CFLAGS="-DSQLITE_ENABLE_FTS5" CGO_LDFLAGS="-lm" go test ./... 2>&1 | grep -v "no test files" |
| | | git add README.md scripts/e2e-verify.sh |
| | | git commit -m "docs: e2e 验证脚本 + README 更新(增量同步/explore/构建要求)" |
| | | git push |
| | | ``` |
| | | |
| | | --- |
| | | |
| | | ## Self-Review |
| | | |
| | | **Spec 覆盖检查:** |
| | | - §1.1 RWR → Task 5 Step 1-5 ✅ |
| | | - §1.2 aliases 进 FTS → Task 1(FTS 表加列)+ Task 5 Step 6-7 ✅ |
| | | - §1.3 status 降权 → Task 5 Step 7 ✅ |
| | | - §2.1 provenance → Task 3 ✅ |
| | | - §2.2 悬空补全 → Task 3(入表)+ Task 4(重试)✅ |
| | | - §3.1 对账 → Task 2 ✅ |
| | | - §3.2 FTS external-content → Task 1 Step 6 ✅ |
| | | - §3.3 schema 迁移 + 备份 → Task 1 Step 4 ✅ |
| | | - §4 explore → Task 6 ✅ |
| | | - §5 config → Task 5 Step 7(Config 加节)+ Task 6(读 config 缺省值)✅ |
| | | - 测试计划 5 项 → Task 1-6 各单测 + Task 7 e2e ✅ |
| | | |
| | | **额外修复(spec 外,实测发现的既有 bug):** CJK 词 FTS5 MATCH 失效 → Task 5 的 LIKE 通道。此为搜索质量层的必要组成部分,已在 spec 的"搜索质量提升"目标内。 |
| | | |
| | | **类型一致性:** `graph.Edge.Provenance`(Task 3 定义,Task 4/5 使用);`index.FileStat`(Task 2 定义);`search.ExploreOptions`(Task 6 定义);`vault.FileMeta.Status/Aliases`(Task 2 定义,Task 3 的 graph.Node 对齐)。`RetryUnresolved` 签名 Task 2 占位 `() {}` → Task 4 改 `(int, error)`——Task 2 的 reconcile 调用处写 `store.RetryUnresolved()` 不接收返回值,兼容两种签名。✅ |
| | | |
| | | **Placeholder 扫描:** 无 TBD/TODO;所有代码步骤含完整代码。Task 2 Step 5 的 `resolveWikilink` 与 Task 3 的 `matchWikilink` 是两套实现(DB 版/内存版),规则一致,注释已说明。✅ |