Author SHA1 Message Date
lyun bfb91efc2b 文档:记录分步加载发布与线上验收 2026-11-14 12:44:51 +08:00
lyun 374f08c526 优化:查询按范围分步加载并精简表格聚合 2026-11-14 12:34:50 +08:00
lyun 381901184c 文档:记录筛选提速与包含条件的仓库交付 2026-11-13 22:16:57 +08:00
lyun 6983dc32eb 同步:来源筛选提速与包含条件 2026-11-13 22:15:58 +08:00
lyun f47341f3f0 Document contains filter usage and live acceptance 2026-11-13 22:10:50 +08:00
lyun f8432b4e02 Bound contains keyword input and clear duplicate search drafts 2026-11-13 21:51:31 +08:00
lyun 3c022e8614 Add literal contains filters across source queries and analysis context 2026-11-13 21:23:12 +08:00
lyun 7cae4cfbdf Record accepted filter-speed release and measured index storage 2026-11-13 20:32:16 +08:00
lyun 20f1c7d0e2 Seek the first visible source row without scanning all fact ids 2026-11-13 20:14:51 +08:00
lyun fa70dc6504 Cover source candidate scope checks without wide fact row reads 2026-11-13 19:19:18 +08:00
lyun 7bd9e51004 优化:按批次重置筛选索引临时窗口 2026-11-13 18:37:11 +08:00
lyun 5e02c6ad66 优化:筛选索引按暂存窗口读取字典 2026-11-13 18:20:32 +08:00
lyun 7782e33bec 优化:统一来源筛选索引与后台构建 2026-11-13 17:56:18 +08:00
lyun 5068b8fbc9 文档:记录来源筛选修复的线上验收 2026-11-13 16:56:38 +08:00
lyun ea84e26b2f 修复:来源字段筛选超时与重复扫描 2026-11-13 16:52:27 +08:00
lyun 5457f5c786 同步:查询钻取修复、不良率细分与预汇总提速 2026-11-13 15:31:45 +08:00
lyun f579934f8b docs: verify compact defect query latency and result equality online 2026-11-13 14:51:46 +08:00
lyun a4eabeb7f2 perf: preaggregate compact defect category statistics for product drilldown 2026-11-13 14:46:55 +08:00
lyun 92b1c88b31 docs: record compact drilldown heading release 2026-11-13 14:21:47 +08:00
lyun 79edb3f6db fix: move defect drilldown breadcrumb beside trend heading 2026-11-13 14:20:15 +08:00
lyun e3e38dec04 docs: verify faster defect trend rendering in production 2026-11-13 14:17:40 +08:00
lyun f8f7335831 fix: display defect trends without waiting for all product details 2026-11-13 14:16:05 +08:00
lyun 696c3d2f6a docs: record defect drilldown release and live acceptance 2026-11-13 13:44:49 +08:00
lyun 63bf8c16d3 feat: drill defect trends into detailed rates and preserve query context 2026-11-13 13:37:32 +08:00
lyun 48ae58e577 文档:整理项目规则与当前开发说明 2026-11-12 22:00:52 +08:00
lyun bc19475b29 修复:DSH 自主查询、图片直显与报告稳定性
允许 Agent 在循环中查目录、修正查询和提问,保留实时权限检查及模型原始回答。
统一相同查询证据与图表资源;排名补齐阶段指标,修正 OR 重量筛选,并限制模型明细输入。
增加运行恢复、图表自适应坐标轴与两位小数,修复报告主图选择、重复图及边缘标签。

验证:独立副本通过 162 项 Python、73 项前端及 21 项后端测试,前端构建通过。
线上已用原 8 题复测,并追加两轮双期对比;TOP100 与组合筛选分别核对 100 行和 20 行。
2026-11-12 16:14:20 +08:00
lyun f1a2fbd494 统一:DSH 智能分析、Top100 与图表静默恢复
接入 DSH 专用运行时和对话界面,保留模型原始回答,支持会话分页、连续追问与显式报告。
补齐组合筛选、各维度 Top100、分组查询到排名图的转换,图表故障静默省略并基于保存数据自动恢复。
优化报告风格、图表顺序和表格换行,更新接入说明及构建输入。

验证:独立目录前后端构建通过;118 项 Java、102 项前端、102 项图表、14 项 DSH 应用和 13 项组合筛选检查通过。
仅提交相关源码、锁定依赖与测试;不包含凭据、会话、业务数据或本地产物。
2026-11-12 11:57:41 +08:00
lyun cc8832c8e5 修复:独立回收中断备份并限制 AI 日志增长 2026-11-10 16:02:46 +08:00
lyun dc5b9b76ec 新增:Linux Docker 部署与自动回收、备份及容量监测 2026-11-10 14:53:07 +08:00
lyun 3962521a5c 修复:加固 Windows 更新预检、提交点和可恢复的数据库回退流程 2026-11-05 16:54:30 +08:00
lyun 2315c83067 修复:Windows 更新备份使用写句柄刷盘,保留失败阶段诊断 2026-11-05 15:06:16 +08:00
lyun 4843768694 修复:Windows 更新检查仅探测程序路径,避免慢磁盘校验超时 2026-11-05 14:40:33 +08:00
lyun b208f93945 修复:旧浏览器样式与安全操作编号兼容,复用已完成的 Windows 数据库恢复 2026-11-05 14:08:09 +08:00
lyun c2410c4ce1 修复:Windows 离线启动误校验 Python 可再生缓存,提供原地修复工具 2026-11-05 11:05:37 +08:00
lyun 7357c309dd 界面:表格长文本列统一左对齐,保留名称与属性对齐 2026-11-04 12:45:39 +08:00
lyun 9f282baa95 修复:目标保存后取消读取时自动重试并切换年度 2026-11-04 12:32:26 +08:00
lyun 866ba21bb8 修复:清理导入历史失败提示并识别旧文件年份 2026-11-04 12:06:48 +08:00
lyun f04d4e96e3 完善:同月多文件导入与当前用户离线迁移更新 2026-11-04 11:35:24 +08:00
lyun 147b56d0fd 修复:快照合并分析图并支持导出订单对应表
快照将不良构成与柏拉图合并为一个选项,完整保留页面容器,黑片和亮片继续并排展示两张分析图;迁移旧快照选项并等待组合图的数据准备完成。

销售订单类型对应表支持下载当前生效版本的完整Excel,订单号按文本写入,保留前导零,沿用管理员角色与来源字段权限。导出开始时固定版本,使用滚动工作簿限制内存。

菜单、页面及快照标题统一改为黑片数据和亮片数据。

验证:11项前端检查、8项后端单元测试通过;157698条实际对应关系与数据库逐条一致,重新导入预览零变更;黑片完整快照和亮片单模块快照均通过视觉检查。
2026-11-04 09:06:07 +08:00
lyun 80cd9e79eb 调整:交换页面快照与查询状态的位置 2026-11-04 08:39:49 +08:00
lyun c3354c1f64 新增:查询页面快照并修复筛选选框闪烁
页面快照支持勾选查询条件、指标卡片、趋势、汇总表、不良构成和细分缺陷图,复制PNG或下载;增加取消、分阶段提示和等待超时恢复。

修复公共Select弹层准备帧被opacity覆盖导致的先闪出再展开,并取消查询字段原生label的重复点击转发。

同步已经验收上线的销售订单类型映射、排除筛选及预聚合,以及其线上权限、删除确认配套源码;保留既有公式编辑和名录编码修改。源码范围按发布清单冻结,不纳入本机数据、凭据及未发布工作。

验证:前后端独立源码构建通过;10项前端检查、4项订单分类单元测试通过;实际系统剪贴板粘贴PNG验证通过;线上配置及筛选结果与发布前一致,后端未重启。
2026-11-04 00:15:32 +08:00
lyun 54813accaa 优化:整理指标目标并支持中文字段公式编辑
黑片和亮片按结果展示完整公式,在同一窗口维护条件与未匹配取值。支持中文字段选择、运算符和求和,并保留原计算语义及特殊策略。

移除重复账号入口与配置说明,统一紧凑表格。保留仓库既有权限判断,不包含此前未同步的角色迁移、预览快照或运行数据。

验证:前端构建及68项相关检查通过。
2026-11-03 16:31:23 +08:00
lyun 1aeccddd4d 优化:名录编码自动生成并隐藏维护入口 2026-11-03 13:11:00 +08:00
lyun 52d92c95d8 修复:智能分析空回答与工具结束条件冲突
统一自主分析工具范围与执行前校验,严格核验正常结束、最终正文和图表恢复状态,并显示空回答及恢复失败原因。

包含两次已上线修复及回归测试,保留已有回答与业务数据。

验证:1101 项 Python、469 项前端、97 项后端测试通过;前后端构建和真实模型、页面验收通过。
2026-11-03 08:56:49 +08:00
485 changed files with 49502 additions and 3356 deletions
+47
View File
@@ -0,0 +1,47 @@
# 仅允许源码和构建输入进入上下文,避免把本机数据库、备份、密码或业务表格送入构建器。
**
!.dockerignore
!backend/
!backend/pom.xml
!backend/src/
!backend/src/**
!web/
!web/package.json
!web/pnpm-lock.yaml
!web/pnpm-workspace.yaml
!web/index.html
!web/tsconfig*.json
!web/vite.config.ts
!web/src/
!web/src/**
!web/public/
!web/public/**
!scripts/
!scripts/**
!ai/
!ai/**
!deploy/
!deploy/docker/
!deploy/docker/**
!deploy/macos/
!deploy/macos/**
!runtime.lock.json
!requirements-nao.in
!requirements-nao.lock
!runtime/
!runtime/dsh/
!runtime/dsh/package.json
!runtime/dsh/package-lock.json
!runtime/dsh/stream-bridge.mjs
!runtime/dsh/application/
!runtime/dsh/application/**
!runtime/dsh/vendor/
!runtime/dsh/vendor/**
**/__pycache__
**/*.pyc
**/.env
**/.env.*
**/secrets
**/original-workbook-acceptance.json
**/*.log
!deploy/docker/.env.example
+5
View File
@@ -5,6 +5,9 @@
/runtimedata/ /runtimedata/
/output/ /output/
/web/node_modules/ /web/node_modules/
/runtime/dsh/node_modules/
/runtime/dsh/bin/
/runtime/dsh/installation.json
/web/dist/ /web/dist/
/backend/target/ /backend/target/
/.env.local /.env.local
@@ -16,6 +19,8 @@ __pycache__/
/verification/* /verification/*
!/verification/test_*.py !/verification/test_*.py
!/verification/*.test.mjs !/verification/*.test.mjs
!/verification/isolated_runtime.py
!/verification/benchmark_scale.py
/设计预览/ /设计预览/
/开发交付说明.md /开发交付说明.md
/无代码数据操作说明.md /无代码数据操作说明.md
+31
View File
@@ -0,0 +1,31 @@
# 质量数据分析平台项目规则
沿用全局沟通与执行习惯;本文件只补充项目约束和入口。
## 项目约束
- 正式指标由后端指标服务和当前有效规则计算。月、季、年先汇总分量再计算,不平均月率;缺失阶段、零分母、去本部报废等语义以正式服务为准,不在页面、导出或 AI 中另造算法。
- 当前智能分析固定使用 DSH,保留模型原始正文;普通提问直接回答,报告仅在明确要求时生成。图表失败静默省略并保留后台诊断。遗留 `nao_*.py` 与根目录兼容入口不表示可以恢复双引擎或增加模型审阅。
- 权限、数据范围及来源绑定由服务端执行;查询、缓存、图表、报告和下载均保留当前授权检查,历史快照不能绕过实时权限。
- 测试使用隔离数据库与合成数据,复用 `OwnedTestDatabase` 或 `scripts/managed_test_resources.py` 登记和清理自有资源。私有原表回归另选入口,不把业务主库当测试库。
- `.runtime/`、`runtimedata/`、`output/` 含私有数据、验收产物及可能正在使用的运行包,不按目录名称判断可删除。密钥、登录资料、业务原件和备份不得进入 Git 或普通交接记录。
- 主工作目录可能在旧分支上积累已由其他工作目录交付的改动。继续开发前核对分支、最新主线及本次差异;不将全部未提交内容一并提交,不切换或重置当前目录来“清理”。
## 按需阅读与代码入口
| 任务 | 入口 |
| --- | --- |
| 接续当前工作 | `docs/当前进度.md`;先核实记录与实际状态 |
| 启动、构建和验证 | `README.md`;共享服务维护另读 `deploy/macos/README.md` |
| 查询、指标和导出 | `web/src/QueryPage.tsx`、`web/src/metricWorkbook.ts`;后端 `metrics/`、`analytics/`,均位于 `backend/src/main/java/cn/yunsheng/quality/` |
| 智能分析 | `docs/DeepSeekHarness正式接入.md`;前端 `web/src/agent/`、后端 `analysis/`、`scripts/build_dsh_application.py` 及 `runtime/dsh/application/` |
| 数据接入和规则 | 后端 `source/`、`ingestion/`、`workflow/`、`masterdata/`、`dictionary/`;本地授权资产存在时,原表核对另读 `verification/rules.md`,其中样本范围属于历史基线,该文件不随源码同步 |
| 界面与交互 | `DESIGN.md` 及现有组件、主题;无需为纯业务改动读取全部设计说明 |
| 备份、回收和迁移 | `deploy/macos/README.md`、`deploy/docker/README.md`;版本变化触发的维护保护须经兼容性核实后再解除 |
## 验证入口
- 前端按影响选择 `node --test web/verification/<相关测试>.test.mjs`;查询导出对应 `metric_workbook.test.mjs`。必要时再执行 `pnpm --dir web build`。
- 完整本机构建入口为 `python3 scripts/project.py build`,运行后端非私有原表测试并构建前端;原表全量回归显式加 `--private-acceptance`,资产缺失不能当作通过。
- Python 用项目对应环境按需运行 `PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=scripts python3 -m unittest discover -s verification -p '<相关测试文件>'`;DSH 环境检查入口为 `python3 scripts/setup_dsh_runtime.py --check`,不调用模型。
- 不因文档变更触发构建、服务重启、模型请求、数据库迁移或维护清理。开发测试通过、仓库同步和线上生效分别记录。
+38
View File
@@ -135,3 +135,41 @@
- 实现与验证详见 docs/全系统统一加载实现与验收_20261102.md。 - 实现与验证详见 docs/全系统统一加载实现与验收_20261102.md。
- 2026-11-02:趋势折线使用用户选定的 A 轻度平滑(0.15),沿横轴约束曲线;查询合格率、不良率、智能分析与导出统一。柏拉图累计线保持直线,数据点、缺失断点和计算口径不变。 - 2026-11-02:趋势折线使用用户选定的 A 轻度平滑(0.15),沿横轴约束曲线;查询合格率、不良率、智能分析与导出统一。柏拉图累计线保持直线,数据点、缺失断点和计算口径不变。
## 2026-11-03:指标与目标(本地预览)
- 采用用户确认的“指标说明、质量目标、计算设置”顺序,默认显示中文指标与当前生效公式。
- 指标说明的修改公式动作定位到同一计算设置表单;名称、别名与说明独立编辑。
- 目标默认表格查看,表单修改年度、范围和百分比;展示既有匹配优先级得出的沿用值,保持保存目标的现有接口与即时生效行为。
- 计算设置默认从黑片计算开始,黑片、亮片各一张表,每个结果只占一行,各自从1开始编号,不分组、不重复展示完整执行顺序;完整公式包含默认值和全部条件分支,长内容使用共享悬浮全文。来源字段收进高级设置,仅用于旧数据兼容。保留原始行数组、所有公式及特殊处理。
- 英文变量放入表单参考详情;率值1解释为100%。延续现有主题、控件、弹窗及切换过渡,不新增位移动效。
- 本次预览使用配置只读快照,保存及失败重试在预览中模拟;未发布线上,未修改计算后端及正式配置。
- 用户反馈行高偏大:指标说明、质量目标、计算设置统一复用现有管理表格的紧凑样式(表头34px、内容行36px),不单独覆盖字号、行距或单元格间距;长内容通过共享悬浮全文与详情查看。
- 调整顺序以完整结果及其分支为单位,只发生在当前阶段内,限制合计早于分量等依赖冲突;编辑在同一窗口维护默认取值、条件和各自特殊策略,分支同时命中时上方优先。编辑与保存前检查不可用变量、重复冲突条件及指标前向引用。校验只作用于维护操作,不改写或求值正式公式;来源变量按对应阶段的服务端字段字典显示中文。
- 2026-11-03:用户确认“一个结果一条完整公式”,本地将黑片13条执行规则呈现为8个结果、亮片9条执行规则呈现为7个结果。相同取值的条件可用“或”展示,底层不合并求值、不改优先级;未修改直接应用须保持原22条规则、位置、数值类型和空值策略完全一致。原有表头34px、行高36px及共享悬浮全文保持。未发布线上。
## 2026-11-03:中文公式编辑与账号入口精简(本地)
- 黑片、亮片及合格率公式共用中文公式编辑器,使用 `[中文字段名]` 引用;可搜索选择字段、在光标处插入、替换选中文字、插入四则运算、括号、逗号和求和。也接受直接输入中文字段名。
- 字段列表区分来源字段和计算结果;重名字段用中文类别消歧,条件用文本字段不进入数值公式列表。合格率只提供已计算的前序指标。
- 保存前将中文引用转换为原有引擎字段,检查结构、参数数量、嵌套与长度,不在浏览器求值,不引入新的计算语言或改变条件与空值策略。不改动时保存须保留原公式文字及全部元数据。
- 删除左下角重复账号区域,保留右上角个人账户菜单及原有资料、密码、退出操作。
- 沿用现有表格密度、主题、弹窗与浮层风格,未发布线上。
- 条件公式编辑按阅读顺序展示:先列条件分支和满足时取值,再列“以上条件均不满足时取值”;无条件结果仍显示“计算公式”。仅调整表单展示顺序,底层初始化位置、条件优先级和特殊策略不变。
- 计算设置移除“配置说明”折叠区和说明编辑框,仅保留隐藏字段以沿用既有保存数据;公式维护和保存入口不变。
- 2026-11-03:本轮指标与目标、完整结果公式、中文字段编辑、条件优先布局、重复账号与配置说明精简,已按用户授权发布为 `20261103-082040Z-rules-workspace`。分母策略的条件隐藏已撤销,线上保持始终显示;正式计算、后端与业务数据未改动。68项检查和线上桌面/窄屏验收通过。
## 2026-11-13:不良率分类钻取
- 点击有细项的分类曲线进入该分类,保留总废率淡虚线、细项曲线及“全部分类”返回;不良率曲线和横轴取消月份筛选,其他趋势保留原交互。
- 表格局部展开当前阶段的分类和细项,另一阶段收拢为关键指标,Excel 跟随当前列;返回恢复原字段和重量/废率选择。
- 年份和产品变化保留分类,阶段变化退出。加载或失败不混用旧范围行与新细项,完成前禁止导出。
- 缺失与零区分,无细项类别不可钻取;覆盖不一致时仅给出简短提示。
- 2026-11-13:不良率钻取“全部分类 › 当前分类”移至趋势标题旁,不再在筛选项与图例之间单独占一行;所有查询页面共用。
+35 -2
View File
@@ -9,15 +9,26 @@
- 公网入口:[https://quality.lyunlab.me](https://quality.lyunlab.me)。服务器提供前端静态页面和 HTTPS;API 经 SSH 加密回连到本机后端。 - 公网入口:[https://quality.lyunlab.me](https://quality.lyunlab.me)。服务器提供前端静态页面和 HTTPS;API 经 SSH 加密回连到本机后端。
- MySQL、模型凭据、业务来源文件和运行数据保存在本机,不包含在本仓库中。本机需保持开机联网且项目服务运行,其他电脑才能使用完整功能。 - MySQL、模型凭据、业务来源文件和运行数据保存在本机,不包含在本仓库中。本机需保持开机联网且项目服务运行,其他电脑才能使用完整功能。
## 开发入口
- [项目规则](AGENTS.md):本项目的业务约束、代码入口和验证选择,配合全局使用习惯执行。
- [当前进度](docs/当前进度.md):最近核实的代码、验证、部署状态及待办;跨会话继续时先核对实际状态。
- [智能分析接入](docs/DeepSeekHarness正式接入.md):当前 DSH、模型原文、报告、图表与平台边界。
- `开发交付说明.md` 及带日期的实施、验收文档保留为历史证据;其中早期数据范围、运行方式和功能状态不能直接视作当前状态。本地交付说明及被 Git 排除的验收资料不随源码同步。
## 本机运行 ## 本机运行
Linux 容器部署、自动重启、空间回收、备份保留及容量告警见 [Docker 部署说明](deploy/docker/README.md)。这些配置随源码进入 Gitea;构建镜像使用手动入口,不要求配置 Gitea 自动构建。Mac 测试服务的开机启动与守护见 [本机服务说明](deploy/macos/README.md)。
本机启动脚本面向 macOS 15 及以上的 Apple Silicon。先准备 Python 3.11 及以上、Node.js/pnpm 和 [uv](https://docs.astral.sh/uv/getting-started/installation/)。首次运行需要安装项目运行依赖并构建: 本机启动脚本面向 macOS 15 及以上的 Apple Silicon。先准备 Python 3.11 及以上、Node.js/pnpm 和 [uv](https://docs.astral.sh/uv/getting-started/installation/)。首次运行需要安装项目运行依赖并构建:
python3 scripts/setup_runtime.py python3 scripts/setup_runtime.py
pnpm --dir web install --frozen-lockfile pnpm --dir web install --frozen-lockfile
python3 scripts/project.py build python3 scripts/project.py build
随后运行: 上述安装、构建及下列启动命令用于独立开发环境。当前共享测试服务已由守护程序运行;日常检查使用 `python3 deploy/macos/service.py status`。维护共享服务前先阅读本机服务说明,不在端口已被占用时重复启动,旧停止命令也不能替代守护管理。
独立开发环境随后运行:
python3 scripts/project.py start python3 scripts/project.py start
@@ -33,7 +44,29 @@
在系统设置中配置本机模型。设置保存在私有的 `.runtime/nao-model-config.json`;旧版根目录 `apikey.txt` 只作为未配置时的兼容入口,严禁提交任一凭据文件。 在系统设置中配置本机模型。设置保存在私有的 `.runtime/nao-model-config.json`;旧版根目录 `apikey.txt` 只作为未配置时的兼容入口,严禁提交任一凭据文件。
AI 助理使用固定的 Python 3.11.15、nao-core 0.3.26 和带哈希的 `requirements-nao.lock`。首次运行依赖安装命令会一并安装 AI 环境;已有环境只检查,不覆盖正在运行的会话。可单独运行 `python3 scripts/setup_nao_runtime.py --check` 检查版本、导入和原生服务器。检查不会调用模型。明确不使用 AI 时,可使用 `setup_runtime.py --without-ai`;需要 AI 时再运行 `setup_nao_runtime.py`。 项目保留固定的 Python 3.11.15 和旧 nao 依赖安装入口;`requirements-nao.lock` 中的 nao-core 0.3.26 属于兼容源码环境,不代表当前正式分析引擎。总安装器仍会准备这些依赖及 DSH;`--without-ai` 仅准备 Java/数据库环境,`--without-dsh` 不是可用的正式智能分析部署方式。已有环境不得为更新说明而覆盖或中断运行中的会话。
质量智能分析固定使用 DeepSeek Harness(DSH)。API 配置管理模型、端点和思考设置,不提供 NAO/DSH 切换。Java 管理身份、授权、正式指标及历史,DSH 执行工具循环;模型正文原样保存,普通问题直接回答,只有明确要求时生成报告。图表失败时静默省略,后台保留诊断。
DSH CLI 固定为 `0.2.2-rc.1`,依赖与官方 Python SDK 校验记录在 `runtime/dsh/`,要求 Node 24.9+。可运行 `python3 scripts/setup_dsh_runtime.py --node /path/to/node24` 安装项目内运行环境,`--check` 检查真实启动且不调用模型。`scripts/build_dsh_application.py` 构建独立应用,正式运行包使用其中的 `scripts/quality_analysis_runtime.py`;根目录同名源码仍有双引擎兼容逻辑,不能据此推断线上支持切换。当前系统隔离与正式服务在 macOS 验证;Linux/Windows 等效隔离未验收时明确拒绝 DSH 启动。详见[正式接入说明](docs/DeepSeekHarness正式接入.md)。
## 不良率趋势钻取
总体、产业部、产品、黑片和亮片查询默认进入年度。切换年份保留当前产品层级;明确指定期间的链接继续使用该期间。
在不良率趋势图中点击有细项的分类曲线,可查看该分类下各细项的废率趋势;分类总废率以虚线保留。点击“全部分类”返回。切换年份或产品保留分类,切换黑片/亮片退出分类。不良率图不再通过点击曲线或横轴筛选月份,合格率与产量图保留原交互。
底部表格同步展开当前分类的细项,重量/废率切换与 Excel 导出跟随当前显示;产品和期间行维度保持不变。细项使用正式阶段同期总重量作分母,月、季、年均先汇总分量再计算;没有来源或分母时显示缺失,真实零显示零。细项与正式分类总量不一致时提示覆盖不足,不补造分类。
## 筛选索引与查询加载
来源字段支持等于、包含、排除和空值筛选;“包含”输入关键词后按回车生效,多个关键词任一命中,区分大小写,`%`、`_` 按普通文字匹配,每个关键词最多256字符、最多50个。各类筛选共用按构建冻结的值字典及记录索引;汇总、趋势、来源明细和导出沿用当前授权与正式指标计算。源字段值保留空值、大小写及前导零语义,旧配置筛选的字符串 `null` 约定不变。下拉框使用只读候选模式,只取一页及是否还有;需要精确总数的原接口继续保留。
启动时在单线程后台补建当前完整正式版本的索引,导入及归类在事务提交后进入同一去重队列。完整构建标记提交后才启用;未完成、失败或未补建的历史构建沿用原始查询,不阻止启动和正式数据发布。索引增加持久存储及后台构建成本,规模取决于非空字段数和不同值数量;高基数宽表应先测容量。保留清理随无引用构建回收索引;升级后原维护兼容保护继续生效。
年度全产品查询先显示正式指标和同范围趋势,再读取产品表格;月/季、单产品及总体/产业部等范围复用完整结果,避免额外往返。不同范围的趋势独立加载,底部柏拉图接近视口且主查询就绪后读取。刷新和筛选变化会使旧结果失去当前资格;表格失败时,已就绪的指标与趋势仍可使用。
普通查询在数据版本已解析后开始,来源字段条件仍等待对应版本的字段目录;产品文本搜索等待必要产品选项,不因无关元数据失败而禁用。表格仍保留全量产品及全量 Excel 导出。
## 旧版本时间迁移 ## 旧版本时间迁移
@@ -2,6 +2,8 @@
先读 /RULES.md 了解正式公式、单位、范围和来源语义;需要时读 /agent/mcps/quality/quality_data.json 一次了解工具。所有业务数据经原生 mcp_call 调用 quality 的 quality_data 取得。不调用数据库、execute_sql、display_chart 或 story;平台负责图表与报告。 先读 /RULES.md 了解正式公式、单位、范围和来源语义;需要时读 /agent/mcps/quality/quality_data.json 一次了解工具。所有业务数据经原生 mcp_call 调用 quality 的 quality_data 取得。不调用数据库、execute_sql、display_chart 或 story;平台负责图表与报告。
本模式开放只读文件工具 read/list/grep/search 和 quality 的 mcp_call/mcp_connect。平台负责查询前的任务澄清、数据绑定和后续追问;不使用原生 clarification、数据库选择、凭据收集、建议追问或写入工具结束任务。证据足够时,在最后一次查询返回后直接输出最终正文,并正常结束;工具返回、图表安排或进度说明不等于最终回答。
本会话使用自主调查模式。遵守RULES中的正式公式、数据语义与证据边界,不设固定全维度覆盖、维度顺序或自动补查配方。以本轮 _agentTask 的明确目标和约束为准:必要基线→观察结果→选择下次查询或验证反例→判断是否已回答。不要为了覆盖目录而查询每个产业部、分组或规格,不固定只追一个产品或一个阶段。若多个线索同样重要,应比较后再判断。 本会话使用自主调查模式。遵守RULES中的正式公式、数据语义与证据边界,不设固定全维度覆盖、维度顺序或自动补查配方。以本轮 _agentTask 的明确目标和约束为准:必要基线→观察结果→选择下次查询或验证反例→判断是否已回答。不要为了覆盖目录而查询每个产业部、分组或规格,不固定只追一个产品或一个阶段。若多个线索同样重要,应比较后再判断。
正式数值直接来自工具,不能自行重算KPI或编造缺失值。你可以解释分子分母、规模、结构及阶段的关系,清楚区分已核实事实、统计线索与待验证生产假设;引用真实 evidenceRef。工具支持的正式差值和阶段拆解可以直接引用;缺少的数据不得补零。统计关联不能宣称已证实设备、工艺、供应商或人员根因。 正式数值直接来自工具,不能自行重算KPI或编造缺失值。你可以解释分子分母、规模、结构及阶段的关系,清楚区分已核实事实、统计线索与待验证生产假设;引用真实 evidenceRef。工具支持的正式差值和阶段拆解可以直接引用;缺少的数据不得补零。统计关联不能宣称已证实设备、工艺、供应商或人员根因。
@@ -30,10 +30,10 @@ public class ManagementController {
} }
@GetMapping("/tasks") @GetMapping("/tasks")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR','DATA_MAINTAINER','DATA_PUBLISHER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> tasks(@RequestParam(defaultValue = "50") int limit,@AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> tasks(@RequestParam(defaultValue = "50") int limit,@AuthenticationPrincipal AppUserPrincipal actor) {
if (limit < 1 || limit > 200) throw new IllegalArgumentException("limit 必须为 1 到 200"); if (limit < 1 || limit > 200) throw new IllegalArgumentException("limit 必须为 1 到 200");
boolean all=actor.getRoleCodes().contains("SYSTEM_ADMIN")||actor.getRoleCodes().contains("SYSTEM_OPERATOR"); boolean all=actor.getRoleCodes().contains("SYSTEM_ADMIN");
List<Map<String, Object>> items = jdbc.query(""" List<Map<String, Object>> items = jdbc.query("""
SELECT id, task_type, status, stage, progress, processed, total, payload, result, SELECT id, task_type, status, stage, progress, processed, total, payload, result,
error_code, error_message, created_at, updated_at,retry_count,max_retries,next_retry_at,retry_parent_id,heartbeat_at error_code, error_message, created_at, updated_at,retry_count,max_retries,next_retry_at,retry_parent_id,heartbeat_at
@@ -44,10 +44,10 @@ public class ManagementController {
/** Read-only summaries; execution endpoints and their permissions remain independent. */ /** Read-only summaries; execution endpoints and their permissions remain independent. */
@GetMapping("/tasks/overview") @GetMapping("/tasks/overview")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR','DATA_MAINTAINER','DATA_PUBLISHER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> taskOverview(@AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> taskOverview(@AuthenticationPrincipal AppUserPrincipal actor) {
boolean allTasks=actor.getRoleCodes().contains("SYSTEM_ADMIN")||actor.getRoleCodes().contains("SYSTEM_OPERATOR"); boolean allTasks=actor.getRoleCodes().contains("SYSTEM_ADMIN");
boolean allWorkflows=actor.getRoleCodes().contains("SYSTEM_ADMIN")||actor.getRoleCodes().contains("DATA_PUBLISHER"); boolean allWorkflows=actor.getRoleCodes().contains("SYSTEM_ADMIN")||actor.getRoleCodes().contains("DATA_ADMIN");
String owner=actor.getId().toString(); String owner=actor.getId().toString();
List<Map<String,Object>> items=new java.util.ArrayList<>(jdbc.query(""" List<Map<String,Object>> items=new java.util.ArrayList<>(jdbc.query("""
SELECT id,task_type,status,stage,progress,processed,total,payload,error_code,error_message,created_at,updated_at SELECT id,task_type,status,stage,progress,processed,total,payload,error_code,error_message,created_at,updated_at
@@ -83,7 +83,7 @@ public class ManagementController {
} }
@GetMapping("/rules") @GetMapping("/rules")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','RULE_MAINTAINER','QUALITY_VIEWER','BUSINESS_ANALYST','AUDIT_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> rules() { public Map<String, Object> rules() {
List<Map<String, Object>> items = jdbc.query(""" List<Map<String, Object>> items = jdbc.query("""
SELECT id, name, status, description, rule_definition, validation, row_version, created_at SELECT id, name, status, description, rule_definition, validation, row_version, created_at
@@ -104,7 +104,7 @@ public class ManagementController {
} }
@GetMapping("/mappings") @GetMapping("/mappings")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','RULE_MAINTAINER','QUALITY_VIEWER','BUSINESS_ANALYST','AUDIT_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
@org.springframework.transaction.annotation.Transactional(readOnly = true) @org.springframework.transaction.annotation.Transactional(readOnly = true)
public Map<String, Object> mappings() { public Map<String, Object> mappings() {
List<Map<String, Object>> items = jdbc.query(""" List<Map<String, Object>> items = jdbc.query("""
@@ -142,7 +142,7 @@ public class ManagementController {
} }
@GetMapping("/audit") @GetMapping("/audit")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','AUDIT_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN')")
public Map<String, Object> audit(@RequestParam(defaultValue = "1") int page, public Map<String, Object> audit(@RequestParam(defaultValue = "1") int page,
@RequestParam(defaultValue = "50") int pageSize) { @RequestParam(defaultValue = "50") int pageSize) {
if (page < 1 || pageSize < 1 || pageSize > 200) throw new IllegalArgumentException("分页参数无效"); if (page < 1 || pageSize < 1 || pageSize > 200) throw new IllegalArgumentException("分页参数无效");
@@ -171,7 +171,7 @@ public class ManagementController {
public Map<String, Object> users() { public Map<String, Object> users() {
List<UserRow> users = jdbc.query(""" List<UserRow> users = jdbc.query("""
SELECT id, username, display_name, active, force_password_change, row_version, created_at SELECT id, username, display_name, active, force_password_change, row_version, created_at
FROM app_user ORDER BY username FROM app_user WHERE deleted_at IS NULL ORDER BY username
""", (rs, row) -> new UserRow(UUID.fromString(rs.getString("id")), rs.getString("username"), """, (rs, row) -> new UserRow(UUID.fromString(rs.getString("id")), rs.getString("username"),
rs.getString("display_name"), rs.getBoolean("active"), rs.getBoolean("force_password_change"), rs.getString("display_name"), rs.getBoolean("active"), rs.getBoolean("force_password_change"),
rs.getLong("row_version"), rs.getTimestamp("created_at").toInstant().toString())); rs.getLong("row_version"), rs.getTimestamp("created_at").toInstant().toString()));
@@ -21,6 +21,7 @@ import org.springframework.security.crypto.password.PasswordEncoder;
import org.springframework.transaction.annotation.Transactional; import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.bind.annotation.PathVariable; import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.PutMapping; import org.springframework.web.bind.annotation.PutMapping;
import org.springframework.web.bind.annotation.RequestBody; import org.springframework.web.bind.annotation.RequestBody;
@@ -31,8 +32,6 @@ import org.springframework.web.server.ResponseStatusException;
@RestController @RestController
@RequestMapping("/api") @RequestMapping("/api")
public class ManagementMutationController { public class ManagementMutationController {
private static final Set<String> ROLES = Set.of("SYSTEM_ADMIN", "SYSTEM_OPERATOR", "DATA_MAINTAINER", "DATA_PUBLISHER",
"RULE_MAINTAINER", "QUALITY_VIEWER", "BUSINESS_ANALYST", "AUDIT_VIEWER");
private final JdbcTemplate jdbc; private final JdbcTemplate jdbc;
private final JsonMapper json; private final JsonMapper json;
private final PasswordEncoder encoder; private final PasswordEncoder encoder;
@@ -93,7 +92,7 @@ public class ManagementMutationController {
if (!targetWillBeAdmin && (activeOtherAdmins == null || activeOtherAdmins == 0)) throw conflict("至少保留一个启用的系统管理员"); if (!targetWillBeAdmin && (activeOtherAdmins == null || activeOtherAdmins == 0)) throw conflict("至少保留一个启用的系统管理员");
int updated = jdbc.update(""" int updated = jdbc.update("""
UPDATE app_user SET display_name=?,active=?,force_password_change=?,row_version=row_version+1 UPDATE app_user SET display_name=?,active=?,force_password_change=?,row_version=row_version+1
WHERE id=? AND row_version=? WHERE id=? AND row_version=? AND deleted_at IS NULL
""", displayName, request.active(), request.forcePasswordChange(), userId.toString(), request.expectedRowVersion()); """, displayName, request.active(), request.forcePasswordChange(), userId.toString(), request.expectedRowVersion());
if (updated == 0) throw conflict("用户版本已变化,请刷新后重试"); if (updated == 0) throw conflict("用户版本已变化,请刷新后重试");
jdbc.update("DELETE FROM app_user_role WHERE user_id=?", userId.toString()); jdbc.update("DELETE FROM app_user_role WHERE user_id=?", userId.toString());
@@ -106,12 +105,45 @@ public class ManagementMutationController {
return user(userId.toString()); return user(userId.toString());
} }
@DeleteMapping("/users/{id}")
@PreAuthorize("hasRole('SYSTEM_ADMIN')")
@Transactional
public Map<String, Object> deleteUser(@PathVariable String id, @RequestBody UserDeleteRequest request,
@AuthenticationPrincipal AppUserPrincipal actor) {
UUID userId = parseId(id);
if (request.expectedRowVersion() == null || request.expectedRowVersion() < 0) throw bad("expectedRowVersion 为必填");
if (userId.equals(actor.getId())) throw conflict("不能删除当前登录账号");
// Use the same lock order as account editing so concurrent changes cannot remove all administrators.
jdbc.query("""
SELECT u.id FROM app_user u JOIN app_user_role r ON r.user_id=u.id
WHERE u.active=TRUE AND r.role_code='SYSTEM_ADMIN' FOR UPDATE
""", (rs, row) -> rs.getString(1));
List<Map<String, Object>> targets = jdbc.query("""
SELECT username,row_version FROM app_user WHERE id=? AND deleted_at IS NULL FOR UPDATE
""", (rs, row) -> Map.<String, Object>of("username", rs.getString("username"), "rowVersion", rs.getLong("row_version")), userId.toString());
if (targets.isEmpty()) throw new ResponseStatusException(HttpStatus.NOT_FOUND, "用户不存在或已删除");
Map<String, Object> target = targets.getFirst();
if (((Number) target.get("rowVersion")).longValue() != request.expectedRowVersion()) throw conflict("用户版本已变化,请刷新后重试");
Integer otherAdmins = jdbc.queryForObject("""
SELECT COUNT(*) FROM app_user u JOIN app_user_role r ON r.user_id=u.id
WHERE u.active=TRUE AND r.role_code='SYSTEM_ADMIN' AND u.id<>?
""", Integer.class, userId.toString());
if (otherAdmins == null || otherAdmins == 0) throw conflict("至少保留一个启用的系统管理员");
jdbc.update("""
UPDATE app_user SET active=FALSE,deleted_at=CURRENT_TIMESTAMP(6),row_version=row_version+1
WHERE id=? AND row_version=? AND deleted_at IS NULL
""", userId.toString(), request.expectedRowVersion());
audit(actor.getId().toString(), "USER_DELETED", "app_user", userId.toString(),
Map.of("username", target.get("username"), "historyRetained", true));
return Map.of("deleted", true, "id", userId.toString());
}
@GetMapping("/rules/schema") @GetMapping("/rules/schema")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','RULE_MAINTAINER','AUDIT_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_ADMIN')")
public Map<String,Object> ruleSchema(){return cn.yunsheng.quality.ingestion.ConfigurableRuleEngine.schema();} public Map<String,Object> ruleSchema(){return cn.yunsheng.quality.ingestion.ConfigurableRuleEngine.schema();}
@PostMapping("/rules") @PostMapping("/rules")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN')")
@Transactional @Transactional
public Map<String, Object> createRule(@RequestBody VersionDraft request, public Map<String, Object> createRule(@RequestBody VersionDraft request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
@@ -125,7 +157,7 @@ public class ManagementMutationController {
} }
@PutMapping("/rules/{id}") @PutMapping("/rules/{id}")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN')")
@Transactional @Transactional
public Map<String, Object> updateRule(@PathVariable String id, @RequestBody VersionDraft request, public Map<String, Object> updateRule(@PathVariable String id, @RequestBody VersionDraft request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
@@ -142,7 +174,7 @@ public class ManagementMutationController {
} }
@PostMapping("/rules/{id}/validate") @PostMapping("/rules/{id}/validate")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN')")
@Transactional @Transactional
public Map<String, Object> validateRule(@PathVariable String id, @RequestBody VersionAction request, public Map<String, Object> validateRule(@PathVariable String id, @RequestBody VersionAction request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
@@ -159,7 +191,7 @@ public class ManagementMutationController {
} }
@PostMapping("/rules/{id}/activate") @PostMapping("/rules/{id}/activate")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN')")
@Transactional @Transactional
public Map<String, Object> activateRule(@PathVariable String id, @RequestBody VersionAction request, public Map<String, Object> activateRule(@PathVariable String id, @RequestBody VersionAction request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
@@ -186,7 +218,7 @@ public class ManagementMutationController {
} }
@PostMapping("/mappings") @PostMapping("/mappings")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
@Transactional @Transactional
public Map<String, Object> createMapping(@RequestBody VersionDraft request, public Map<String, Object> createMapping(@RequestBody VersionDraft request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
@@ -201,7 +233,7 @@ public class ManagementMutationController {
} }
@PutMapping("/mappings/{id}") @PutMapping("/mappings/{id}")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
@Transactional @Transactional
public Map<String, Object> updateMapping(@PathVariable String id, @RequestBody VersionDraft request, public Map<String, Object> updateMapping(@PathVariable String id, @RequestBody VersionDraft request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
@@ -219,7 +251,7 @@ public class ManagementMutationController {
} }
@PostMapping("/mappings/{id}/validate") @PostMapping("/mappings/{id}/validate")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
@Transactional @Transactional
public Map<String, Object> validateMapping(@PathVariable String id, @RequestBody VersionAction request, public Map<String, Object> validateMapping(@PathVariable String id, @RequestBody VersionAction request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
@@ -244,7 +276,7 @@ public class ManagementMutationController {
} }
@PostMapping("/mappings/{id}/activate") @PostMapping("/mappings/{id}/activate")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
@Transactional @Transactional
public Map<String, Object> activateMapping(@PathVariable String id, @RequestBody VersionAction request, public Map<String, Object> activateMapping(@PathVariable String id, @RequestBody VersionAction request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
@@ -314,10 +346,7 @@ public class ManagementMutationController {
} }
private List<String> validateRoles(List<String> values) { private List<String> validateRoles(List<String> values) {
if (values == null || values.isEmpty()) throw bad("至少选择一个角色"); return cn.yunsheng.quality.identity.AppRoles.validate(values);
List<String> roles = values.stream().distinct().sorted().toList();
if (!ROLES.containsAll(roles)) throw bad("包含不支持的角色");
return roles;
} }
private List<String> normalizeScopes(List<String> values, String kind) { private List<String> normalizeScopes(List<String> values, String kind) {
@@ -449,6 +478,7 @@ public class ManagementMutationController {
List<String> departments, List<String> groups) {} List<String> departments, List<String> groups) {}
public record UserUpdateRequest(String displayName, boolean active, boolean forcePasswordChange, List<String> roles, public record UserUpdateRequest(String displayName, boolean active, boolean forcePasswordChange, List<String> roles,
List<String> departments, List<String> groups, Long expectedRowVersion) {} List<String> departments, List<String> groups, Long expectedRowVersion) {}
public record UserDeleteRequest(Long expectedRowVersion) {}
public record VersionDraft(String id, String name, String description, Object definition, Long expectedRowVersion) {} public record VersionDraft(String id, String name, String description, Object definition, Long expectedRowVersion) {}
public record VersionAction(Long expectedRowVersion,Boolean confirmDifferences,String regressionRunId) { public record VersionAction(Long expectedRowVersion,Boolean confirmDifferences,String regressionRunId) {
public VersionAction(Long expectedRowVersion){this(expectedRowVersion,false,null);} public VersionAction(Long expectedRowVersion){this(expectedRowVersion,false,null);}
@@ -40,14 +40,14 @@ public class TaskActionController {
} }
@GetMapping("/{id}") @GetMapping("/{id}")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR','DATA_MAINTAINER','DATA_PUBLISHER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> get(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> get(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor) {
authorize(id,actor); authorize(id,actor);
return Map.of("task", task(id)); return Map.of("task", task(id));
} }
@GetMapping("/{id}/issues") @GetMapping("/{id}/issues")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR','DATA_MAINTAINER','DATA_PUBLISHER','AUDIT_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> issues(@PathVariable String id,@RequestParam(defaultValue="1") int page,@RequestParam(defaultValue="100") int pageSize,@AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> issues(@PathVariable String id,@RequestParam(defaultValue="1") int page,@RequestParam(defaultValue="100") int pageSize,@AuthenticationPrincipal AppUserPrincipal actor) {
authorize(id,actor);access.requireDetail(actor);if(page<1||pageSize<1||pageSize>500)throw new IllegalArgumentException("分页参数无效"); authorize(id,actor);access.requireDetail(actor);if(page<1||pageSize<1||pageSize>500)throw new IllegalArgumentException("分页参数无效");
Map<String, Object> task = task(id); Map<String, Object> task = task(id);
@@ -72,14 +72,14 @@ public class TaskActionController {
} }
@PostMapping("/{id}/retry") @PostMapping("/{id}/retry")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR','DATA_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> retry(@PathVariable String id, @RequestBody RetryRequest request, public Map<String, Object> retry(@PathVariable String id, @RequestBody RetryRequest request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
return retryService.retry(id,request.idempotencyKey(),request.expectedCurrentReleaseId(),actor); return retryService.retry(id,request.idempotencyKey(),request.expectedCurrentReleaseId(),actor);
} }
@PostMapping("/{id}/cancel") @PostMapping("/{id}/cancel")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR','DATA_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
@Transactional @Transactional
public Map<String, Object> cancel(@PathVariable String id, @RequestBody CancelRequest request, public Map<String, Object> cancel(@PathVariable String id, @RequestBody CancelRequest request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
@@ -123,7 +123,7 @@ public class TaskActionController {
if (rows.isEmpty()) throw new ResponseStatusException(HttpStatus.NOT_FOUND, "任务不存在"); if (rows.isEmpty()) throw new ResponseStatusException(HttpStatus.NOT_FOUND, "任务不存在");
return rows.getFirst(); return rows.getFirst();
} }
private void authorize(String id,AppUserPrincipal actor){String owner=jdbc.queryForObject("SELECT created_by FROM task WHERE id=?",String.class,id);if(!actor.getRoleCodes().contains("SYSTEM_ADMIN")&&!actor.getRoleCodes().contains("SYSTEM_OPERATOR")&&!java.util.Objects.equals(owner,actor.getId().toString()))throw new ResponseStatusException(HttpStatus.FORBIDDEN,"无权访问该任务");} private void authorize(String id,AppUserPrincipal actor){String owner=jdbc.queryForObject("SELECT created_by FROM task WHERE id=?",String.class,id);if(!actor.getRoleCodes().contains("SYSTEM_ADMIN")&&!java.util.Objects.equals(owner,actor.getId().toString()))throw new ResponseStatusException(HttpStatus.FORBIDDEN,"无权访问该任务");}
private Object parse(String value) { private Object parse(String value) {
if (value == null) return Map.of(); if (value == null) return Map.of();
@@ -0,0 +1,21 @@
package cn.yunsheng.quality.analysis;
import cn.yunsheng.quality.common.DepartmentLabels;
import java.util.*;
/** Agent-visible organization directory; historical source data stays intact. */
final class AnalysisAgentCatalog {
static Map<String,Object> active(Map<String,Object> source) {
var result=new LinkedHashMap<>(source);
result.put("departments",AnalysisProtocolService.list(source.get("departments")).stream()
.filter(x->x instanceof String name&&!DepartmentLabels.isRetired(name)).toList());
var owners=new LinkedHashMap<String,Object>();
AnalysisProtocolService.map(source.get("groupDepartments")).forEach((group,department)->{
if(department instanceof String name&&!DepartmentLabels.isRetired(name))owners.put(group,department);
});
result.put("groupDepartments",owners);
result.put("groups",AnalysisProtocolService.list(source.get("groups")).stream()
.filter(group->!AnalysisProtocolService.map(source.get("groupDepartments")).containsKey(group)||owners.containsKey(group)).toList());
return result;
}
}
@@ -61,7 +61,7 @@ public class AnalysisApiSettingsService {
String configured = property != null && !property.isBlank() ? property : environment; String configured = property != null && !property.isBlank() ? property : environment;
if (configured != null && !configured.isBlank()) return Path.of(configured).toAbsolutePath().normalize(); if (configured != null && !configured.isBlank()) return Path.of(configured).toAbsolutePath().normalize();
for (Path p = start.toAbsolutePath().normalize(); p != null; p = p.getParent()) for (Path p = start.toAbsolutePath().normalize(); p != null; p = p.getParent())
if (Files.isRegularFile(p.resolve("scripts/nao_quality_runtime.py"))) return p; if (Files.isRegularFile(p.resolve("scripts/quality_analysis_runtime.py"))) return p;
throw new IllegalStateException("Quality project root was not found"); throw new IllegalStateException("Quality project root was not found");
} }
@@ -70,11 +70,18 @@ public class AnalysisApiSettingsService {
return view(c, profiles(c)); return view(c, profiles(c));
} }
/** Returns the selected analysis engine without exposing any model credentials. */
public synchronized String runtimeEngine() {
return AnalysisRuntimeEngine.DSH;
}
public synchronized Map<String, Object> save(Settings request) { public synchronized Map<String, Object> save(Settings request) {
Config current = load(); Config current = load();
if (request == null || request.revision() == null) throw bad("请先读取当前 API 配置"); if (request == null || request.revision() == null) throw bad("请先读取当前 API 配置");
if (!current.revision().equals(request.revision())) if (!current.revision().equals(request.revision()))
throw new ResponseStatusException(HttpStatus.CONFLICT, "API 配置已被修改,请重新打开后再保存"); throw new ResponseStatusException(HttpStatus.CONFLICT, "API 配置已被修改,请重新打开后再保存");
if (request.runtimeEngine() != null && !AnalysisRuntimeEngine.DSH.equalsIgnoreCase(request.runtimeEngine().strip()))
throw bad("线上分析引擎固定为 DeepSeek Harness");
List<Profile> profiles = profiles(current); List<Profile> profiles = profiles(current);
Profile selected = selectedProfile(request.profileId(), current, profiles); Profile selected = selectedProfile(request.profileId(), current, profiles);
Config next = resolve(request, selected.config()); Config next = resolve(request, selected.config());
@@ -87,11 +94,13 @@ public class AnalysisApiSettingsService {
try { try {
Files.createDirectories(target.getParent()); Files.createDirectories(target.getParent());
if (Files.isSymbolicLink(target.getParent()) || Files.isSymbolicLink(target)) throw new IOException(); if (Files.isSymbolicLink(target.getParent()) || Files.isSymbolicLink(target)) throw new IOException();
temp = Files.createTempFile(target.getParent(), ".nao-model-", ".tmp"); temp = Files.createTempFile(target.getParent(), ".quality-model-", ".tmp");
Files.setPosixFilePermissions(temp, PosixFilePermissions.fromString("rw-------")); if (Files.getFileStore(temp).supportsFileAttributeView("posix")) {
Files.setPosixFilePermissions(temp, PosixFilePermissions.fromString("rw-------"));
}
List<Map<String,Object>> storedProfiles = updated.stream().map(p -> Map.<String,Object>of( List<Map<String,Object>> storedProfiles = updated.stream().map(p -> Map.<String,Object>of(
"id",p.id(),"name",p.name(),"baseUrl",p.config().baseUrl(),"apiKey",p.config().apiKey(),"model",p.config().model(),"thinkingEnabled",p.config().thinkingEnabled(),"thinkingEffort",p.config().thinkingEffort())).toList(); "id",p.id(),"name",p.name(),"baseUrl",p.config().baseUrl(),"apiKey",p.config().apiKey(),"model",p.config().model(),"thinkingEnabled",p.config().thinkingEnabled(),"thinkingEffort",p.config().thinkingEffort())).toList();
Files.writeString(temp, json.writeValueAsString(Map.of("schemaVersion", 3, "baseUrl", next.baseUrl(), Files.writeString(temp, json.writeValueAsString(Map.of("schemaVersion", 4, "baseUrl", next.baseUrl(),
"apiKey", next.apiKey(), "model", next.model(), "revision", next.revision(), "apiKey", next.apiKey(), "model", next.model(), "revision", next.revision(),
"activeProfileId", selected.id(), "profiles", storedProfiles,"thinkingEnabled",next.thinkingEnabled(),"thinkingEffort",next.thinkingEffort())), StandardCharsets.UTF_8); "activeProfileId", selected.id(), "profiles", storedProfiles,"thinkingEnabled",next.thinkingEnabled(),"thinkingEffort",next.thinkingEffort())), StandardCharsets.UTF_8);
Files.move(temp, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING); Files.move(temp, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
@@ -249,8 +258,8 @@ public class AnalysisApiSettingsService {
} }
private Path settingsPath() { private Path settingsPath() {
String selected=System.getenv("QUALITY_NAO_MODEL_CONFIG_FILE"); String selected=System.getenv("QUALITY_ANALYSIS_MODEL_CONFIG_FILE");
return selected==null||selected.isBlank()?root.resolve(".runtime/nao-model-config.json"):Path.of(selected).toAbsolutePath().normalize(); return selected==null||selected.isBlank()?root.resolve(".runtime/quality-model-config.json"):Path.of(selected).toAbsolutePath().normalize();
} }
private List<Profile> profiles(Config current) { private List<Profile> profiles(Config current) {
try { try {
@@ -308,6 +317,8 @@ public class AnalysisApiSettingsService {
private Map<String, Object> view(Config c,List<Profile> profiles) { private Map<String, Object> view(Config c,List<Profile> profiles) {
var result = new LinkedHashMap<String,Object>(); var result = new LinkedHashMap<String,Object>();
result.putAll(Map.of("baseUrl", c.baseUrl(), "model", c.model(), "hasApiKey", !c.apiKey().isBlank(), "revision", c.revision(),"thinkingEnabled",c.thinkingEnabled(),"thinkingEffort",c.thinkingEffort())); result.putAll(Map.of("baseUrl", c.baseUrl(), "model", c.model(), "hasApiKey", !c.apiKey().isBlank(), "revision", c.revision(),"thinkingEnabled",c.thinkingEnabled(),"thinkingEffort",c.thinkingEffort()));
result.put("runtimeEngine", AnalysisRuntimeEngine.DSH);
result.put("runtimeEngineLocked", true);
result.put("maxOutputTokens",AnalysisRuntimePolicy.MAX_OUTPUT_TOKENS); result.put("maxOutputTokens",AnalysisRuntimePolicy.MAX_OUTPUT_TOKENS);
result.put("activeProfileId",activeProfileId(c,profiles)); result.put("activeProfileId",activeProfileId(c,profiles));
result.put("profiles",profiles.stream().map(p->Map.of("id",p.id(),"name",p.name(),"baseUrl",p.config().baseUrl(), result.put("profiles",profiles.stream().map(p->Map.of("id",p.id(),"name",p.name(),"baseUrl",p.config().baseUrl(),
@@ -326,7 +337,10 @@ public class AnalysisApiSettingsService {
return text; return text;
} }
private static ResponseStatusException bad(String message) { return new ResponseStatusException(HttpStatus.BAD_REQUEST, message); } private static ResponseStatusException bad(String message) { return new ResponseStatusException(HttpStatus.BAD_REQUEST, message); }
public record Settings(String baseUrl, String apiKey, String model, String revision, String profileId, Boolean thinkingEnabled, String thinkingEffort) { public record Settings(String baseUrl, String apiKey, String model, String revision, String profileId, Boolean thinkingEnabled, String thinkingEffort, String runtimeEngine) {
public Settings(String baseUrl,String apiKey,String model,String revision,String profileId,Boolean thinkingEnabled,String thinkingEffort) {
this(baseUrl,apiKey,model,revision,profileId,thinkingEnabled,thinkingEffort,null);
}
public Settings(String baseUrl,String apiKey,String model,String revision,String profileId) { this(baseUrl,apiKey,model,revision,profileId,null,null); } public Settings(String baseUrl,String apiKey,String model,String revision,String profileId) { this(baseUrl,apiKey,model,revision,profileId,null,null); }
public Settings(String baseUrl,String apiKey,String model,String revision) { this(baseUrl,apiKey,model,revision,null); } public Settings(String baseUrl,String apiKey,String model,String revision) { this(baseUrl,apiKey,model,revision,null); }
@Override public String toString() { return "Settings[credentials redacted]"; } @Override public String toString() { return "Settings[credentials redacted]"; }
@@ -20,7 +20,7 @@ public class AnalysisChangesController {
} }
@PostMapping("/changes") @PostMapping("/changes")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> query(@RequestBody Map<String, Object> body, public Map<String, Object> query(@RequestBody Map<String, Object> body,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
return changes.query(body, actor); return changes.query(body, actor);
@@ -639,7 +639,7 @@ public class AnalysisChangesService {
String sort = body.containsKey("sort") ? text(body.get("sort"), "sort") : "rate_drop"; String sort = body.containsKey("sort") ? text(body.get("sort"), "sort") : "rate_drop";
if (!SORTS.contains(sort)) throw bad("sort 只支持 rate_drop、current_rate、scrap_delta、current_scrap"); if (!SORTS.contains(sort)) throw bad("sort 只支持 rate_drop、current_rate、scrap_delta、current_scrap");
int limit = body.containsKey("limit") ? integer(body.get("limit"), "limit") : 10; int limit = body.containsKey("limit") ? integer(body.get("limit"), "limit") : 10;
if (limit < 1 || limit > 10) throw bad("limit 必须在 1 到 10 之间"); if (limit < 1 || limit > 100) throw bad("limit 必须在 1 到 100 之间");
List<String> fields = fields(body.get("fields"), body.containsKey("fields")); List<String> fields = fields(body.get("fields"), body.containsKey("fields"));
return new Request(releaseId, current, comparison, filters, dimension, stage, sort, limit, fields); return new Request(releaseId, current, comparison, filters, dimension, stage, sort, limit, fields);
} }
@@ -21,6 +21,7 @@ import java.util.Objects;
import java.util.UUID; import java.util.UUID;
import java.util.Set; import java.util.Set;
import org.springframework.http.HttpStatus; import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.jdbc.core.JdbcTemplate; import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.security.access.prepost.PreAuthorize; import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.security.core.annotation.AuthenticationPrincipal; import org.springframework.security.core.annotation.AuthenticationPrincipal;
@@ -33,15 +34,17 @@ import org.springframework.web.bind.annotation.PatchMapping;
import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody; import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController; import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.server.ResponseStatusException; import org.springframework.web.server.ResponseStatusException;
import org.springframework.web.servlet.mvc.method.annotation.SseEmitter;
import tools.jackson.databind.JsonNode; import tools.jackson.databind.JsonNode;
import tools.jackson.databind.json.JsonMapper; import tools.jackson.databind.json.JsonMapper;
/** AI never computes a KPI or changes data; every tool call uses the shared metric service and current identity. */ /** AI never computes a KPI or changes data; every tool call uses the shared metric service and current identity. */
@RestController @RestController
@RequestMapping("/api/analysis") @RequestMapping("/api/analysis")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','BUSINESS_ANALYST','QUALITY_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public class AnalysisController { public class AnalysisController {
private final MetricsService metrics; private final MetricsService metrics;
private final AnalysisModelGateway model; private final AnalysisModelGateway model;
@@ -52,8 +55,11 @@ public class AnalysisController {
private final TransactionTemplate transaction; private final TransactionTemplate transaction;
private final AnalysisApiSettingsService apiSettings; private final AnalysisApiSettingsService apiSettings;
private AnalysisProtocolService protocol; private AnalysisProtocolService protocol;
private AnalysisTurnService turnService;
@org.springframework.beans.factory.annotation.Autowired @org.springframework.beans.factory.annotation.Autowired
public void setProtocol(AnalysisProtocolService protocol){this.protocol=protocol;} public void setProtocol(AnalysisProtocolService protocol){this.protocol=protocol;}
@org.springframework.beans.factory.annotation.Autowired
public void setTurnService(AnalysisTurnService turnService) { this.turnService = turnService; }
@org.springframework.beans.factory.annotation.Value("${quality.analysis.visuals-enabled:true}") @org.springframework.beans.factory.annotation.Value("${quality.analysis.visuals-enabled:true}")
private boolean visualsEnabled = true; private boolean visualsEnabled = true;
@@ -89,9 +95,9 @@ public class AnalysisController {
String host = URI.create(baseUrl).getHost(); String host = URI.create(baseUrl).getHost();
if (host == null || host.isBlank()) return unconfiguredModelStatus(); if (host == null || host.isBlank()) return unconfiguredModelStatus();
return Map.of("configured", true, "model", configuredModel, "serviceHost", host, return Map.of("configured", true, "model", configuredModel, "serviceHost", host,
"mode", "NAO_AGENT_SERVER_TOOLS", "businessResultsSentToModel", true, "mode", "DSH_AGENT_SERVER_TOOLS", "businessResultsSentToModel", true,
"authorizedQueryResultsMayBeSentToModel", true, "authorizedQueryResultsMayBeSentToModel", true,
"capabilities", List.of("nativeNaoRuntime", "authorizedServerTools", "structuredQuery", "capabilities", List.of("nativeDshRuntime", "authorizedServerTools", "structuredQuery",
"statistics", "periodicReports")); "statistics", "periodicReports"));
} catch (RuntimeException invalidOrUnavailable) { } catch (RuntimeException invalidOrUnavailable) {
return unconfiguredModelStatus(); return unconfiguredModelStatus();
@@ -99,7 +105,7 @@ public class AnalysisController {
} }
private static Map<String, Object> unconfiguredModelStatus() { private static Map<String, Object> unconfiguredModelStatus() {
return Map.of("configured", false, "mode", "NAO_NOT_CONFIGURED", return Map.of("configured", false, "mode", "DSH_NOT_CONFIGURED",
"businessResultsSentToModel", false, "authorizedQueryResultsMayBeSentToModel", false); "businessResultsSentToModel", false, "authorizedQueryResultsMayBeSentToModel", false);
} }
@@ -147,35 +153,86 @@ public class AnalysisController {
@GetMapping("/sessions/{id}") @GetMapping("/sessions/{id}")
public Map<String, Object> history(@PathVariable String id, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> history(@PathVariable String id, @AuthenticationPrincipal AppUserPrincipal actor) {
return historyWindow(id, actor, null, null);
}
@GetMapping("/sessions/{id}/history")
public Map<String, Object> historyPage(@PathVariable String id,
@RequestParam(required = false) String before,
@AuthenticationPrincipal AppUserPrincipal actor) {
if (before != null && !before.matches("[a-fA-F0-9-]{36}"))
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "历史游标无效");
return historyWindow(id, actor, 6, before);
}
@GetMapping(value = "/sessions/{id}/follow", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public SseEmitter follow(@PathVariable String id, @AuthenticationPrincipal AppUserPrincipal actor) {
// This authorization and the turn-plus-cursor read run before sending any SSE bytes.
Map<String, Object> snapshot = historyWindow(id, actor, 6, null);
return turnService.followSession(snapshot, actor);
}
private record SavedTurn(String id, String question, String answer, String steps, String plan,
String status, BigDecimal created, BigDecimal started, BigDecimal finished, String eventCursor) {}
private Map<String, Object> historyWindow(String id, AppUserPrincipal actor, Integer pageSize, String before) {
governance.requireDetail(actor); governance.requireDetail(actor);
governance.requireBoundAction("QUERY", "query.detail"); governance.requireBoundAction("QUERY", "query.detail");
Session session = ownedSession(id, actor); Session session = ownedSession(id, actor);
// A retained version alone never grants access; re-run the scoped query before returning stored evidence. // A retained version alone never grants access; re-run the scoped query before returning stored evidence.
if(protocol!=null&&protocol.ownsSession(id))protocol.authorizeHistory(id,actor); boolean protocolSession = protocol != null && protocol.ownsSession(id);
if(protocolSession)protocol.authorizeHistory(id,actor);
else metrics.calculate(toQuery(session.context(), session.releaseId()), actor); else metrics.calculate(toQuery(session.context(), session.releaseId()), actor);
Set<String> allowedFields = governance.allowedFields(actor); Set<String> allowedFields = governance.allowedFields(actor);
Set<String> visibleFilterKeys = AnalysisVisualSnapshotAccess.visibleExtraFilterKeys(jdbc, allowedFields); Set<String> visibleFilterKeys = AnalysisVisualSnapshotAccess.visibleExtraFilterKeys(jdbc, allowedFields);
List<Map<String, Object>> turns = jdbc.query("SELECT id,question,answer_json,steps_json,plan_json,created_at,status FROM analysis_turn WHERE session_id=? AND user_id=? ORDER BY created_at", String owner = actor.getId().toString();
(rs, n) -> { var arguments = new ArrayList<Object>(List.of(session.id(), owner));
String turnId = rs.getString("id"); String sql = """
Map<String, Object> plan = rs.getString("plan_json") == null ? Map.of() : read(rs.getString("plan_json")); SELECT id,question,answer_json,steps_json,plan_json,status,
if(protocol!=null&&protocol.ownsSession(id)&&"report".equals(AnalysisProtocolService.map(plan.get("task")).get("kind")) UNIX_TIMESTAMP(created_at)*1000,UNIX_TIMESTAMP(started_at)*1000,UNIX_TIMESTAMP(finished_at)*1000,
&&Set.of("COMPLETED","PARTIAL").contains(Objects.toString(rs.getString("status"),""))) (SELECT MAX(e.id) FROM analysis_turn_event e WHERE e.turn_id=analysis_turn.id)
protocol.authorizeReportSnapshot(turnId,actor); FROM analysis_turn WHERE session_id=? AND user_id=?
if(protocol!=null&&protocol.ownsSession(id)&&AnalysisProtocolInspection.KIND.equals(AnalysisProtocolService.map(plan.get("task")).get("kind")) """;
&&Set.of("COMPLETED","PARTIAL").contains(Objects.toString(rs.getString("status"),"")))protocol.authorizeInspectionSnapshot(turnId,actor); if (before != null) {
Map<String, Object> expectedScope = new LinkedHashMap<>(AnalysisTurnService.retryContext(plan)); sql += " AND (created_at,id) < (SELECT created_at,id FROM analysis_turn WHERE id=? AND session_id=? AND user_id=?)";
expectedScope.remove("_task"); arguments.addAll(List.of(before, session.id(), owner));
var visible=visibleAnswer(rs.getString("answer_json"),turnId,expectedScope,allowedFields,visibleFilterKeys); }
if(visualsEnabled&&protocol!=null&&protocol.ownsTurn(turnId)&&Set.of("COMPLETED","PARTIAL").contains(rs.getString("status"))) sql += pageSize == null ? " ORDER BY created_at,id" : " ORDER BY created_at DESC,id DESC LIMIT ?";
visible.putAll(protocol.visualRecoveryOverlay(turnId,actor)); if (pageSize != null) arguments.add(pageSize + 1);
return Map.of("id", turnId, "question", rs.getString("question"), List<SavedTurn> saved = jdbc.query(sql, (rs, n) -> new SavedTurn(rs.getString("id"),
"result", visible, rs.getString("question"), rs.getString("answer_json"), rs.getString("steps_json"), rs.getString("plan_json"),
"steps", rs.getString("steps_json") == null ? List.of() : mapper.readValue(rs.getString("steps_json"), List.class), rs.getString("status"), rs.getBigDecimal(7), rs.getBigDecimal(8), rs.getBigDecimal(9), rs.getString(10)), arguments.toArray());
"plan", plan, "createdAt", rs.getTimestamp("created_at").toInstant().toString()); boolean hasMore = pageSize != null && saved.size() > pageSize;
}, if (hasMore) saved = saved.subList(0, pageSize);
session.id(), actor.getId().toString()); if (pageSize != null) saved = saved.reversed();
return Map.of("id", session.id(), "title", session.title(), "releaseId", session.releaseId(), "context", session.context(), "turns", turns, "readOnly",protocol!=null&&protocol.legacyReadOnly(id)); List<Map<String, Object>> turns = new ArrayList<>();
for (SavedTurn row : saved) {
Map<String, Object> plan = row.plan() == null ? Map.of() : read(row.plan());
boolean settled = "COMPLETED".equals(row.status()) || "PARTIAL".equals(row.status());
Object taskKind = AnalysisProtocolService.map(plan.get("task")).get("kind");
if (protocolSession && settled && "report".equals(taskKind)) protocol.authorizeReportSnapshot(row.id(), actor);
if (protocolSession && settled && AnalysisProtocolInspection.KIND.equals(taskKind)) protocol.authorizeInspectionSnapshot(row.id(), actor);
Map<String, Object> scope = new LinkedHashMap<>(AnalysisTurnService.retryContext(plan));
scope.remove("_task");
var visible = visibleAnswer(row.answer(), row.id(), scope, allowedFields, visibleFilterKeys);
if (visualsEnabled && settled && protocol != null && protocol.ownsTurn(row.id()))
visible.putAll(protocol.visualRecoveryOverlay(row.id(), actor));
var turn = new LinkedHashMap<String, Object>();
turn.put("id", row.id()); turn.put("question", row.question()); turn.put("status", row.status());
turn.put("result", AnalysisRuntimeEngine.publicMetadata(visible));
turn.put("plan", AnalysisRuntimeEngine.publicMetadata(plan));
turn.put("steps", row.steps() == null ? List.of() : mapper.readValue(row.steps(), List.class));
turn.putAll(AnalysisTurnService.turnTimes(row.created(), row.started(), row.finished()));
if (row.eventCursor() != null) turn.put("eventCursor", row.eventCursor());
if (!AnalysisRuntimeEngine.canExecute(plan)) turn.put("retryAvailable", false);
turns.add(turn);
}
String activeTurnId = turns.stream()
.filter(turn -> "QUEUED".equals(turn.get("status")) || "RUNNING".equals(turn.get("status")))
.map(turn -> (String) turn.get("id")).findFirst().orElse("");
return Map.of("id", session.id(), "title", session.title(), "releaseId", session.releaseId(),
"context", session.context(), "turns", turns, "before", hasMore ? saved.getFirst().id() : "",
"readOnly", protocol != null && protocol.legacyReadOnly(id), "activeTurnId", activeTurnId);
} }
private Map<String, Object> visibleAnswer(String encoded, String turnId, Map<String, Object> expectedScope, private Map<String, Object> visibleAnswer(String encoded, String turnId, Map<String, Object> expectedScope,
@@ -33,7 +33,7 @@ final class AnalysisDirectConversation {
static Map<String,Object> answer(Map<String,Object> task,String release,Map<String,Object> context) { static Map<String,Object> answer(Map<String,Object> task,String release,Map<String,Object> context) {
validate(task); validate(task);
var answer=new LinkedHashMap<String,Object>();answer.put("answer",AnalysisProtocolService.map(task.get("directResponse")).get("answer")); var answer=new LinkedHashMap<String,Object>();answer.put("answer",AnalysisProtocolService.map(task.get("directResponse")).get("answer"));
answer.put("engine","NAO");answer.put("releaseId",release);answer.put("context",context);answer.put("task",task); answer.put("engine",AnalysisRuntimeEngine.DSH);answer.put("releaseId",release);answer.put("context",context);answer.put("task",task);
answer.put("completionMethod","direct_conversation");answer.put("businessResultsSentToModel",false);answer.put("stopReason","direct_conversation"); answer.put("completionMethod","direct_conversation");answer.put("businessResultsSentToModel",false);answer.put("stopReason","direct_conversation");
for(String key:List.of("results","visuals","queryEvidenceRefs","protocolQueryIds"))answer.put(key,List.of()); for(String key:List.of("results","visuals","queryEvidenceRefs","protocolQueryIds"))answer.put(key,List.of());
answer.put("visualSchemaVersion",1);answer.put("visualState","not_requested");return answer; answer.put("visualSchemaVersion",1);answer.put("visualState","not_requested");return answer;
@@ -369,7 +369,7 @@ public class AnalysisExplorationTools {
SelectionValues.appendPredicate(sql, args, "f.product", query.product(), value -> value); SelectionValues.appendPredicate(sql, args, "f.product", query.product(), value -> value);
SelectionValues.appendPredicate(sql, args, "f.project_group", query.projectGroup(), value -> value); SelectionValues.appendPredicate(sql, args, "f.project_group", query.projectGroup(), value -> value);
appendDetailExclusions(sql, args, query); appendDetailExclusions(sql, args, query);
new QueryFilterFields(jdbc,governance).append(sql,args,stage,query.extraFilters(),actor); new QueryFilterFields(jdbc,governance).append(sql,args,stage,query.extraFilters(),actor,releaseId);
DataScope.append(sql, args, actor, "f.department COLLATE utf8mb4_0900_as_cs", DataScope.append(sql, args, actor, "f.department COLLATE utf8mb4_0900_as_cs",
"f.group_name COLLATE utf8mb4_0900_as_cs"); "f.group_name COLLATE utf8mb4_0900_as_cs");
boolean includeUnassigned = query.includeUnassigned() boolean includeUnassigned = query.includeUnassigned()
@@ -23,7 +23,7 @@ public class AnalysisFineDefectsController {
} }
@PostMapping @PostMapping
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','BUSINESS_ANALYST','QUALITY_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> query(@RequestBody(required = false) Map<String, Object> request, public Map<String, Object> query(@RequestBody(required = false) Map<String, Object> request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
governance.requireDetail(actor); governance.requireDetail(actor);
@@ -122,6 +122,7 @@ public class AnalysisFineDefectsService {
&& isAllowed(category.componentCode(), allowed)) componentCodes.add(category.componentCode()); && isAllowed(category.componentCode(), allowed)) componentCodes.add(category.componentCode());
} }
Aggregator aggregate = new Aggregator(request, visibleFields, categories, componentCodes, json); Aggregator aggregate = new Aggregator(request, visibleFields, categories, componentCodes, json);
Aggregator complement = exclusionComplement(request, input, releaseId, actor, allowed, visibleFields, categories, componentCodes, predicate, args);
if (rollups != null && request.dimension().equals("defect") && request.extraFilters().isEmpty() if (rollups != null && request.dimension().equals("defect") && request.extraFilters().isEmpty()
&& (!(input.get("columnFilters") instanceof List<?> filters) || filters.isEmpty()) && (!(input.get("columnFilters") instanceof List<?> filters) || filters.isEmpty())
&& rollups.read(aggregate, request, allowed, predicate.toString(), args)) { && rollups.read(aggregate, request, allowed, predicate.toString(), args)) {
@@ -150,7 +151,112 @@ public class AnalysisFineDefectsService {
} }
return null; return null;
}); });
return withScope(aggregate.response(request, release), scopeContext); if (complement != null) complement.subtract(aggregate);
return withScope((complement == null ? aggregate : complement).response(request, release), scopeContext);
}
/** One authorized scan (or rollup query) for every period/object and all visible children. */
@Transactional(readOnly = true, isolation = Isolation.REPEATABLE_READ)
public Map<String, Object> grouped(Map<String, Object> input, AppUserPrincipal actor, String dimension, String granularity) {
Set<String> allowed = governance.allowedFields(actor);
Request request = parse(input, allowed, true);
String bucket = switch (dimension) {
case "period" -> switch (granularity) {
case "month" -> "f.month_key";
case "quarter" -> "CONCAT(LEFT(f.month_key,4),'-Q',FLOOR((CAST(RIGHT(f.month_key,2) AS UNSIGNED)-1)/3)+1)";
case "year" -> "LEFT(f.month_key,4)";
default -> throw bad("趋势粒度无效");
};
case "overall" -> "'overall'";
case "department" -> "COALESCE(f.department,'__UNASSIGNED__') COLLATE utf8mb4_0900_as_cs";
case "group" -> "COALESCE(f.group_name,'__UNASSIGNED__') COLLATE utf8mb4_0900_as_cs";
case "product" -> "COALESCE(f.product,'__UNASSIGNED__') COLLATE utf8mb4_0900_as_cs";
case "projectGroup" -> "COALESCE(f.project_group,'__UNASSIGNED__') COLLATE utf8mb4_0900_as_cs";
default -> throw bad("细分维度无效");
};
if (request.category() == null) throw bad("必须指定不良分类");
String releaseId = releases.resolvePublishedRelease(request.releaseId());
Map<String, Object> release = releaseContext(releaseId);
if (hasExtendedScope(input) && sourceColumns == null) throw bad("扩展缺陷查询范围当前不可用");
StringBuilder predicate = new StringBuilder(" WHERE rm.release_id = ?");
List<Object> args = new ArrayList<>(List.of(releaseId));
appendScope(predicate,args,request,input,releaseId,actor);
if (request.productSearch() != null) {
predicate.append(" AND LOCATE(LOWER(?),LOWER(COALESCE(f.product,'')))>0"); args.add(request.productSearch());
}
List<DefectField> fields = applicableFields(request,allowed);
List<Category> categories = categories(request.stage());
Set<String> components = new LinkedHashSet<>();
for (Category category : categories) if (fields.stream().anyMatch(f -> f.category().equals(category.name()))
&& isAllowed(category.componentCode(),allowed)) components.add(category.componentCode());
java.util.function.Supplier<Aggregator> create = () -> new Aggregator(request,fields,categories,components,json);
Map<String,Aggregator> groups = new LinkedHashMap<>();
if (!fields.isEmpty()) {
boolean read = rollups != null && request.extraFilters().isEmpty()
&& (!(input.get("columnFilters") instanceof List<?> filters) || filters.isEmpty())
&& rollups.readGroups(groups,create,request,allowed,predicate.toString(),args,bucket,dimension);
if (!read) {
List<String> rawKeys = new ArrayList<>(fields.stream().map(DefectField::key).toList());
categories.stream().map(c -> ExpectedSourceDetailPolicy.totalField(c.componentCode()))
.filter(java.util.Objects::nonNull).filter(k -> isAllowed(k,allowed)).forEach(rawKeys::add);
String sql = "SELECT " + bucket + " AS bucket_key," + jsonObject("f.raw_values",rawKeys)
+ " AS raw_fields," + jsonObject("f.components",new ArrayList<>(components))
+ " AS formal_components,f.group_name AS business_group FROM release_member rm "
+ "JOIN transform_build b ON b.id=rm.transform_build_id AND b.source_revision_id=rm.source_revision_id "
+ "JOIN source_revision sr ON sr.id=rm.source_revision_id JOIN fact_" + request.stage()
+ " f ON f.build_id=rm.transform_build_id AND f.source_revision_id=rm.source_revision_id " + predicate;
jdbc.query(connection -> {
var statement=connection.prepareStatement(sql,ResultSet.TYPE_FORWARD_ONLY,ResultSet.CONCUR_READ_ONLY);
statement.setFetchSize(Integer.MIN_VALUE); new ArgumentPreparedStatementSetter(args.toArray()).setValues(statement);return statement;
},(ResultSetExtractor<Void>) rs -> {
while(rs.next()) groups.computeIfAbsent(rs.getString("bucket_key"),key -> create.get())
.add(readObject(rs.getString("raw_fields")),readObject(rs.getString("formal_components")),null,rs.getString("business_group"));
return null;
});
}
}
Map<String,Object> result = new LinkedHashMap<>();
groups.forEach((key,value) -> {
String responseKey=dimension.equals("department")&&!key.equals("__UNASSIGNED__")&&!key.isBlank()?DepartmentLabels.display(key):key;
if(result.containsKey(responseKey))throw bad("产业部名称映射冲突,暂不能计算细分废率");
result.put(responseKey,value.response(request,release));
});
return mapOf("releaseId",releaseId,"fields",fields.stream().map(f -> mapOf("key",f.key(),"label",f.label(),"category",f.category())).toList(),"groups",result);
}
/** Exact counters allow subtracting a small excluded set without rescanning all retained raw JSON. */
private Aggregator exclusionComplement(Request request, Map<String,Object> input, String releaseId, AppUserPrincipal actor,
Set<String> allowed, List<DefectField> fields, List<Category> categories, Set<String> components,
StringBuilder predicate, List<Object> args) {
String key = cn.yunsheng.quality.filters.SalesOrderTypes.EXCLUDE_KEY;
if (rollups == null || !request.dimension().equals("defect") || !request.extraFilters().keySet().equals(Set.of(key))
|| (input.get("columnFilters") instanceof List<?> filters && !filters.isEmpty())) return null;
var typed = new cn.yunsheng.quality.metrics.SalesOrderMetricRollups(jdbc);
if (!typed.ready(releaseId, request.extraFilters(), false, allowed)) return null;
var baseInput = new LinkedHashMap<>(input); baseInput.remove("extraFilters");
Request base = parse(baseInput, allowed);
var baseWhere = new StringBuilder(" WHERE rm.release_id = ?");
var baseArgs = new ArrayList<Object>(); baseArgs.add(releaseId);
appendScope(baseWhere, baseArgs, base, baseInput, releaseId, actor);
if (request.productSearch() != null) { baseWhere.append(" AND LOCATE(LOWER(?), LOWER(COALESCE(f.product,''))) > 0"); baseArgs.add(request.productSearch()); }
var countWhere = new StringBuilder(baseWhere); var countArgs = new ArrayList<>(baseArgs);
countWhere.append(" AND f.stage=?"); countArgs.add(request.stage());
typed.append(countWhere, countArgs, Map.of());
String countSql = "SELECT COALESCE(SUM(f.source_row_count),0) FROM release_member rm JOIN sales_order_metric_rollup f ON f.build_id=rm.transform_build_id";
Long total = jdbc.queryForObject(countSql+countWhere, Long.class, countArgs.toArray());
var retainedWhere = new StringBuilder(countWhere); var retainedArgs = new ArrayList<>(countArgs);
typed.append(retainedWhere, retainedArgs, request.extraFilters());
Long retained = jdbc.queryForObject(countSql+retainedWhere, Long.class, retainedArgs.toArray());
if (total == null || retained == null || total-retained > retained) return null;
var all = new Aggregator(request, fields, categories, components, json);
if (!rollups.read(all, request, allowed, baseWhere.toString(), baseArgs)) return null;
var selectedInput = new LinkedHashMap<>(input);
selectedInput.put("extraFilters",Map.of(cn.yunsheng.quality.filters.SalesOrderTypes.KEY,request.extraFilters().get(key)));
Request selected = parse(selectedInput, allowed);
predicate.setLength(0); predicate.append(" WHERE rm.release_id = ?"); args.clear(); args.add(releaseId);
appendScope(predicate, args, selected, selectedInput, releaseId, actor);
if (request.productSearch() != null) { predicate.append(" AND LOCATE(LOWER(?), LOWER(COALESCE(f.product,''))) > 0"); args.add(request.productSearch()); }
return all;
} }
private static boolean hasExtendedScope(Map<String, Object> input) { private static boolean hasExtendedScope(Map<String, Object> input) {
@@ -170,7 +276,7 @@ public class AnalysisFineDefectsService {
SelectionValues.appendPredicate(predicate, args, "f.project_group", request.projectGroup(), value -> value); SelectionValues.appendPredicate(predicate, args, "f.project_group", request.projectGroup(), value -> value);
if (!request.includeUnassigned() && !SelectionValues.parse(request.group()).contains("__UNASSIGNED__")) if (!request.includeUnassigned() && !SelectionValues.parse(request.group()).contains("__UNASSIGNED__"))
predicate.append(" AND (f.group_name IS NOT NULL AND f.group_name <> '')"); predicate.append(" AND (f.group_name IS NOT NULL AND f.group_name <> '')");
new QueryFilterFields(jdbc, governance).append(predicate, args, request.stage(), request.extraFilters(), actor); new QueryFilterFields(jdbc, governance).append(predicate, args, request.stage(), request.extraFilters(), actor,releaseId);
DataScope.append(predicate, args, actor, "f.department COLLATE utf8mb4_0900_as_cs", DataScope.append(predicate, args, actor, "f.department COLLATE utf8mb4_0900_as_cs",
"f.group_name COLLATE utf8mb4_0900_as_cs"); "f.group_name COLLATE utf8mb4_0900_as_cs");
} }
@@ -192,6 +298,10 @@ public class AnalysisFineDefectsService {
} }
static Request parse(Map<String, Object> input, Set<String> allowedFields) { static Request parse(Map<String, Object> input, Set<String> allowedFields) {
return parse(input, allowedFields, false);
}
static Request parse(Map<String, Object> input, Set<String> allowedFields, boolean grouped) {
if (input == null) throw bad("请求体不能为空"); if (input == null) throw bad("请求体不能为空");
for (String key : input.keySet()) if (!REQUEST_KEYS.contains(key)) throw bad("未知请求属性:" + key); for (String key : input.keySet()) if (!REQUEST_KEYS.contains(key)) throw bad("未知请求属性:" + key);
if (hasExtendedScope(input)) SourceColumnQueries.normalizedScope(input); if (hasExtendedScope(input)) SourceColumnQueries.normalizedScope(input);
@@ -206,7 +316,7 @@ public class AnalysisFineDefectsService {
if (month != null && (month < 1 || month > 12)) throw bad("月份无效"); if (month != null && (month < 1 || month > 12)) throw bad("月份无效");
if (quarter != null && (quarter < 1 || quarter > 4)) throw bad("季度无效"); if (quarter != null && (quarter < 1 || quarter > 4)) throw bad("季度无效");
if (month != null && quarter != null) throw bad("月份与季度筛选不可同时指定"); if (month != null && quarter != null) throw bad("月份与季度筛选不可同时指定");
if (mode.equals("top") && year == null) throw bad("top 查询必须指定 year"); if (mode.equals("top") && year == null && !grouped) throw bad("top 查询必须指定 year");
String stage = optionalText(input, "stage"); String stage = optionalText(input, "stage");
if (stage != null && !Set.of("black", "bright").contains(stage)) throw bad("stage 必须是 black 或 bright"); if (stage != null && !Set.of("black", "bright").contains(stage)) throw bad("stage 必须是 black 或 bright");
if (mode.equals("top") && stage == null) throw bad("top 查询必须指定 stage"); if (mode.equals("top") && stage == null) throw bad("top 查询必须指定 stage");
@@ -502,6 +612,19 @@ public class AnalysisFineDefectsService {
} }
} }
void subtract(Aggregator excluded) {
if (excluded.sourceRows > sourceRows) throw new IllegalStateException("排除行数超过来源行数");
sourceRows -= excluded.sourceRows;
for (var entry : excluded.fieldStats.entrySet()) subtractStats(fieldStats.get(entry.getKey()), entry.getValue());
for (var entry : excluded.componentStats.entrySet()) subtractStats(componentStats.get(entry.getKey()), entry.getValue());
excluded.expectedRows.forEach((key,value)->expectedRows.merge(key,-value,Long::sum));
excluded.expectedWeights.forEach((key,value)->expectedWeights.merge(key,value.negate(),BigDecimal::add));
}
private static void subtractStats(FieldStats total, FieldStats excluded) {
total.sum=total.sum.subtract(excluded.sum); total.presentRows-=excluded.presentRows;
total.missingRows-=excluded.missingRows; total.invalidRows-=excluded.invalidRows; total.negativeRows-=excluded.negativeRows;
}
Map<String, Object> response(Request request, Map<String, Object> release) { Map<String, Object> response(Request request, Map<String, Object> release) {
List<Map<String, Object>> rows = new ArrayList<>(request.dimension().equals("defect") ? defectRows() : productRows()); List<Map<String, Object>> rows = new ArrayList<>(request.dimension().equals("defect") ? defectRows() : productRows());
rows.sort(Comparator.<Map<String, Object>, BigDecimal>comparing(row -> (BigDecimal) row.get("weight"), rows.sort(Comparator.<Map<String, Object>, BigDecimal>comparing(row -> (BigDecimal) row.get("weight"),
@@ -18,7 +18,7 @@ public class AnalysisInvestigationController {
this.access = access; this.access = access;
} }
@PostMapping @PostMapping
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','BUSINESS_ANALYST','QUALITY_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String,Object> query(@RequestBody Map<String,Object> request, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String,Object> query(@RequestBody Map<String,Object> request, @AuthenticationPrincipal AppUserPrincipal actor) {
// Bound service checks can lock their identity row; keep them outside the read-only scan transaction. // Bound service checks can lock their identity row; keep them outside the read-only scan transaction.
access.requireBoundAction("QUERY", "query.detail"); access.requireBoundAction("QUERY", "query.detail");
@@ -90,7 +90,7 @@ public class AnalysisInvestigationService {
SelectionValues.appendPredicate(predicate,args,"f.product",request.product(),x->x); SelectionValues.appendPredicate(predicate,args,"f.product",request.product(),x->x);
SelectionValues.appendPredicate(predicate,args,"f.project_group",request.projectGroup(),x->x); SelectionValues.appendPredicate(predicate,args,"f.project_group",request.projectGroup(),x->x);
if(!request.includeUnassigned()&&!SelectionValues.parse(request.group()).contains("__UNASSIGNED__"))predicate.append(" AND (f.group_name IS NOT NULL AND f.group_name<>'')"); if(!request.includeUnassigned()&&!SelectionValues.parse(request.group()).contains("__UNASSIGNED__"))predicate.append(" AND (f.group_name IS NOT NULL AND f.group_name<>'')");
new QueryFilterFields(jdbc,access).append(predicate,args,request.stage(),request.extraFilters(),actor); new QueryFilterFields(jdbc,access).append(predicate,args,request.stage(),request.extraFilters(),actor,release);
DataScope.append(predicate,args,actor,"f.department COLLATE utf8mb4_0900_as_cs","f.group_name COLLATE utf8mb4_0900_as_cs"); DataScope.append(predicate,args,actor,"f.department COLLATE utf8mb4_0900_as_cs","f.group_name COLLATE utf8mb4_0900_as_cs");
} }
if(request.productSearch()!=null){predicate.append(" AND LOCATE(?, f.product)>0");args.add(request.productSearch());} if(request.productSearch()!=null){predicate.append(" AND LOCATE(?, f.product)>0");args.add(request.productSearch());}
@@ -14,7 +14,7 @@ final class AnalysisProtocolChanges {
static Map<String,Object> request(String release,Map<String,Object> child) { static Map<String,Object> request(String release,Map<String,Object> child) {
Map<String,Object> query=map(child.get("changeQuery")); Map<String,Object> query=map(child.get("changeQuery"));
if(!query.keySet().equals(Set.of("dimension","sort","limit"))||!Set.of("group","department","product").contains(query.get("dimension")) if(!query.keySet().equals(Set.of("dimension","sort","limit"))||!Set.of("group","department","product").contains(query.get("dimension"))
||!SORTS.contains(query.get("sort"))||count(query.get("limit"),-1)<1||count(query.get("limit"),-1)>10) ||!SORTS.contains(query.get("sort"))||count(query.get("limit"),-1)<1||count(query.get("limit"),-1)>100)
throw invalid("变化节点缺少明确的分组、排序或展示上限"); throw invalid("变化节点缺少明确的分组、排序或展示上限");
List<?> stages=list(child.get("stages")); List<?> stages=list(child.get("stages"));
if(stages.size()>1||!Set.of("black","bright").containsAll(stages))throw invalid("变化节点仅支持综合或单个阶段"); if(stages.size()>1||!Set.of("black","bright").containsAll(stages))throw invalid("变化节点仅支持综合或单个阶段");
@@ -1,11 +1,44 @@
package cn.yunsheng.quality.analysis; package cn.yunsheng.quality.analysis;
import java.util.*; import java.util.*;
import cn.yunsheng.quality.common.SelectionValues;
import tools.jackson.databind.json.JsonMapper; import tools.jackson.databind.json.JsonMapper;
/** Compiles one observed next step without changing the confirmed business task. */ /** Compiles one observed next step without changing the confirmed business task. */
final class AnalysisProtocolExploration { final class AnalysisProtocolExploration {
static boolean nativeAgent(Map<String,Object> task){return "native_agent".equals(task.get("executionMode"));} static boolean nativeAgent(Map<String,Object> task){return "native_agent".equals(task.get("executionMode"));}
/** 启动仅绑定执行身份;业务名称和参数由每次工具调用确认。 */
static Map<String,Object> start(Map<String,Object> proposed) {
var task=new LinkedHashMap<>(proposed);
task.put("kind","analysis");task.put("filters",Map.of());task.put("needsClarification",false);
task.remove("queryPlan");task.remove("clarificationRequest");task.remove("directResponse");
return task;
}
/** 可变业务查询范围不能改变执行身份、发布版本或真实账号权限。 */
static Map<String,Object> withFrame(Map<String,Object> root,Map<String,Object> frame,Map<String,Object> catalog) {
if(!Boolean.TRUE.equals(root.get("agentDriven")))throw new IllegalArgumentException("当前执行未启用自主参数修正");
if(!Set.of("period","comparisonPeriod","filters").containsAll(frame.keySet())
||!(frame.get("period") instanceof Map<?,?>)||!(frame.get("filters") instanceof Map<?,?>))
throw new IllegalArgumentException("查询范围需要period对象和filters对象;请修正后重新调用工具");
var filters=AnalysisProtocolService.map(frame.get("filters"));
if(!Set.of("department","group","product","projectGroup","extraFilters","exclusions","includeUnassigned","columnFilters").containsAll(filters.keySet()))
throw new IllegalArgumentException("查询包含未支持的筛选字段,请先读取catalog");
for(var item:Map.of("department","departments","group","groups","product","products","projectGroup","projectGroups").entrySet()) {
Object value=filters.get(item.getKey());
if(value!=null) {
List<String> names;
try {names=SelectionValues.parse(Objects.toString(value));}
catch(RuntimeException invalid){throw new IllegalArgumentException("筛选值格式不正确,请读取catalog并使用返回的完整name");}
if(names.isEmpty()||!AnalysisProtocolService.list(catalog.get(item.getValue())).containsAll(names))
throw new IllegalArgumentException("筛选名称未匹配当前目录,请用catalog检索候选后修正或向用户提问:"+item.getKey());
}
}
var result=new LinkedHashMap<>(root);result.put("period",frame.get("period"));result.put("filters",filters);
result.put("stages",List.of());result.remove("comparisonPeriod");
if(frame.get("comparisonPeriod")!=null)result.put("comparisonPeriod",frame.get("comparisonPeriod"));
return result;
}
static AnalysisProtocolQueryPlan.Resolution compile(JsonMapper json,Map<String,Object> root,Map<String,Object> node, static AnalysisProtocolQueryPlan.Resolution compile(JsonMapper json,Map<String,Object> root,Map<String,Object> node,
Map<String,Object> inherited,String execution,String release,Map<String,Object> catalog) { Map<String,Object> inherited,String execution,String release,Map<String,Object> catalog) {
String purpose=Objects.toString(node.get("purpose"),""); String purpose=Objects.toString(node.get("purpose"),"");
@@ -38,7 +71,7 @@ final class AnalysisProtocolExploration {
} }
var task=new LinkedHashMap<>(root);task.put("filters",selected);task.put("queryPlan",List.of(next)); var task=new LinkedHashMap<>(root);task.put("filters",selected);task.put("queryPlan",List.of(next));
var dimensions=new LinkedHashSet<Object>(); var dimensions=new LinkedHashSet<Object>();
if(Set.of("ranking","changes").contains(purpose))dimensions.add(node.get("dimension")); if(Set.of("ranking","changes").contains(purpose)||"metrics".equals(purpose)&&node.containsKey("dimension"))dimensions.add(node.get("dimension"));
if("defects".equals(purpose)){dimensions.add("defect");if("product".equals(node.get("dimension")))dimensions.add("product");} if("defects".equals(purpose)){dimensions.add("defect");if("product".equals(node.get("dimension")))dimensions.add("product");}
task.put("dimensions",List.copyOf(dimensions));task.put("fields",Set.of("source","records").contains(purpose)?List.of():node.get("fields")); task.put("dimensions",List.copyOf(dimensions));task.put("fields",Set.of("source","records").contains(purpose)?List.of():node.get("fields"));
task.put("stages",next.get("stages"));task.put("trendRequired","trend".equals(purpose)); task.put("stages",next.get("stages"));task.put("trendRequired","trend".equals(purpose));
@@ -891,7 +891,7 @@ final class AnalysisProtocolHistoryFacts {
String label=text(columnLabels.get(field),historyFieldLabel(section,sources,field)); String label=text(columnLabels.get(field),historyFieldLabel(section,sources,field));
if(label.isBlank())label="授权列"; if(label.isBlank())label="授权列";
List<String> prefix=strings(rule.get("stages")).stream().map(AnalysisProtocolHistoryFacts::stageLabel).toList(); List<String> prefix=strings(rule.get("stages")).stream().map(AnalysisProtocolHistoryFacts::stageLabel).toList();
String operator=switch(text(rule.get("operator"),"")){case "in"->"包含";case "not_in"->"不含";case "is_empty"->"为空";case "not_empty"->"非空";default->"筛选";}; String operator=switch(text(rule.get("operator"),"")){case "in"->"包含";case "contains"->"包含文字";case "not_in"->"不含";case "is_empty"->"为空";case "not_empty"->"非空";default->"筛选";};
List<String> values=scopeLiterals(rule.get("values")); List<String> values=scopeLiterals(rule.get("values"));
StringBuilder part=new StringBuilder();if(!prefix.isEmpty())part.append(String.join("、",prefix)).append(' '); StringBuilder part=new StringBuilder();if(!prefix.isEmpty())part.append(String.join("、",prefix)).append(' ');
part.append(label).append(' ').append(operator);if(!values.isEmpty())part.append(' ').append(String.join("、",values));parts.add(part.toString()); part.append(label).append(' ').append(operator);if(!values.isEmpty())part.append(' ').append(String.join("、",values));parts.add(part.toString());
@@ -44,7 +44,6 @@ final class AnalysisProtocolQueryPlan {
if(stages.isEmpty()&&!fields.equals(List.of("rate"))&&!changesPurpose&&!combinedRanking)throw invalid("重量和报废率节点必须明确阶段;合计排名仅支持报废总重及综合合格率"); if(stages.isEmpty()&&!fields.equals(List.of("rate"))&&!changesPurpose&&!combinedRanking)throw invalid("重量和报废率节点必须明确阶段;合计排名仅支持报废总重及综合合格率");
boolean compare=bool(node.getOrDefault("comparisonRequired",false)); boolean compare=bool(node.getOrDefault("comparisonRequired",false));
if(compare&&(!Set.of("metrics","source","records","defects","changes").contains(purpose)||!(task.get("comparisonPeriod") instanceof Map<?,?>)))throw invalid("期间比较节点需要两个已确认期间"); if(compare&&(!Set.of("metrics","source","records","defects","changes").contains(purpose)||!(task.get("comparisonPeriod") instanceof Map<?,?>)))throw invalid("期间比较节点需要两个已确认期间");
if("trend".equals(purpose)&&map(task.get("period")).containsKey("month"))throw invalid("月度趋势需要多个月份范围");
String preference=string(node.getOrDefault("visualPreference",task.getOrDefault("visualPreference","auto"))); String preference=string(node.getOrDefault("visualPreference",task.getOrDefault("visualPreference","auto")));
if(!Set.of("auto","required","none").contains(preference))throw invalid("节点图表偏好无效"); if(!Set.of("auto","required","none").contains(preference))throw invalid("节点图表偏好无效");
Object visual=node.get("visualType"); Object visual=node.get("visualType");
@@ -90,7 +89,7 @@ final class AnalysisProtocolQueryPlan {
case "previous_ranking" -> { case "previous_ranking" -> {
if(!selector.keySet().equals(Set.of("kind","queryId","stage","metric","ranks")))throw invalid("历史排名选择器格式无效"); if(!selector.keySet().equals(Set.of("kind","queryId","stage","metric","ranks")))throw invalid("历史排名选择器格式无效");
String query=string(selector.get("queryId")),stage=string(selector.get("stage")),metric=string(selector.get("metric")); String query=string(selector.get("queryId")),stage=string(selector.get("stage")),metric=string(selector.get("metric"));
List<Integer> ranks=integers(selector.get("ranks"),1,10);count=ranks.size(); List<Integer> ranks=integers(selector.get("ranks"),1,100);count=ranks.size();
Map<String,Object> available=availableRankings.stream().filter(row->query.equals(row.get("queryId"))&&stage.equals(row.get("stage"))&&metric.equals(row.get("metric"))&&"product".equals(row.get("dimension"))).findFirst().orElseThrow(()->invalid("旧排名缺少当前授权的真实快照,请重新查询排名")); Map<String,Object> available=availableRankings.stream().filter(row->query.equals(row.get("queryId"))&&stage.equals(row.get("stage"))&&metric.equals(row.get("metric"))&&"product".equals(row.get("dimension"))).findFirst().orElseThrow(()->invalid("旧排名缺少当前授权的真实快照,请重新查询排名"));
for(int rank:ranks)if(list(available.get("rows")).stream().noneMatch(row->Objects.equals(map(row).get("rank"),rank)))throw invalid("选择序号不在上轮实际展示排名中"); for(int rank:ranks)if(list(available.get("rows")).stream().noneMatch(row->Objects.equals(map(row).get("rank"),rank)))throw invalid("选择序号不在上轮实际展示排名中");
requestedDimensions.add("product");cleanSelector.put("queryId",query);cleanSelector.put("stage",stage);cleanSelector.put("metric",metric);cleanSelector.put("ranks",ranks); requestedDimensions.add("product");cleanSelector.put("queryId",query);cleanSelector.put("stage",stage);cleanSelector.put("metric",metric);cleanSelector.put("ranks",ranks);
@@ -108,7 +107,7 @@ final class AnalysisProtocolQueryPlan {
if("ranking".equals(purpose)) { if("ranking".equals(purpose)) {
String dimension=string(node.get("dimension")); String dimension=string(node.get("dimension"));
if(!DIMENSIONS.contains(dimension)||!(combinedRanking||WEIGHTS.containsAll(fields)&&!stages.isEmpty())||compare||!kind.equals("base"))throw invalid("排名节点需要正式重量和基础分组;合计口径使用空阶段"); if(!DIMENSIONS.contains(dimension)||!(combinedRanking||WEIGHTS.containsAll(fields)&&!stages.isEmpty())||compare||!kind.equals("base"))throw invalid("排名节点需要正式重量和基础分组;合计口径使用空阶段");
requestedDimensions.add(dimension);clean.put("dimension",dimension);clean.put("limit",integer(node.getOrDefault("limit",10),1,10)); requestedDimensions.add(dimension);clean.put("dimension",dimension);clean.put("limit",integer(node.getOrDefault("limit",10),1,100));
} else if(changesPurpose) { } else if(changesPurpose) {
String dimension=string(node.get("dimension")),sort=string(node.get("sort")); String dimension=string(node.get("dimension")),sort=string(node.get("sort"));
if(!DIMENSIONS.contains(dimension)||!AnalysisProtocolChanges.SORTS.contains(sort)||stages.size()>1||!compare if(!DIMENSIONS.contains(dimension)||!AnalysisProtocolChanges.SORTS.contains(sort)||stages.size()>1||!compare
@@ -116,12 +115,16 @@ final class AnalysisProtocolQueryPlan {
||Set.of("rate_drop","current_rate").contains(sort)&&!fields.contains("rate") ||Set.of("rate_drop","current_rate").contains(sort)&&!fields.contains("rate")
||Set.of("scrap_delta","current_scrap").contains(sort)&&!fields.contains("scrapWeight") ||Set.of("scrap_delta","current_scrap").contains(sort)&&!fields.contains("scrapWeight")
||!Set.of("base","changes_top").contains(kind))throw invalid("变化节点需要两个期间、正式率、综合或单阶段及明确排序"); ||!Set.of("base","changes_top").contains(kind))throw invalid("变化节点需要两个期间、正式率、综合或单阶段及明确排序");
requestedDimensions.add(dimension);clean.put("dimension",dimension);clean.put("sort",sort);clean.put("limit",integer(node.getOrDefault("limit",10),1,10)); requestedDimensions.add(dimension);clean.put("dimension",dimension);clean.put("sort",sort);clean.put("limit",integer(node.getOrDefault("limit",10),1,100));
} else if(defectPurpose) { } else if(defectPurpose) {
if(!fields.equals(List.of("scrapWeight")))throw invalid("缺陷节点只接受来源细项重量"); if(!fields.equals(List.of("scrapWeight")))throw invalid("缺陷节点只接受来源细项重量");
Map<String,Object> query=AnalysisProtocolDefects.normalizeNode(node,stages,catalog); Map<String,Object> query=AnalysisProtocolDefects.normalizeNode(node,stages,catalog);
clean.putAll(query);requestedDimensions.add("defect"); clean.putAll(query);requestedDimensions.add("defect");
if("product".equals(query.get("dimension")))requestedDimensions.add("product"); if("product".equals(query.get("dimension")))requestedDimensions.add("product");
} else if("metrics".equals(purpose)&&node.containsKey("dimension")) {
String dimension=string(node.get("dimension"));
if(!Set.of("department","group","product").contains(dimension)||compare||!kind.equals("base"))throw invalid("批量汇总只支持当前基础范围的产业部、业务分组或产品");
requestedDimensions.add(dimension);clean.put("dimension",dimension);clean.put("limit",integer(node.getOrDefault("limit",100),1,500));
} else if(node.containsKey("dimension")||node.containsKey("limit"))throw invalid("指标或趋势节点不能伪装为分组排名"); } else if(node.containsKey("dimension")||node.containsKey("limit"))throw invalid("指标或趋势节点不能伪装为分组排名");
int periodCount=compare?2:1; int periodCount=compare?2:1;
int readsPerScope=changesPurpose||combinedRanking?1:"ranking".equals(purpose)?fields.size()*stages.size():rawPurpose||defectPurpose?stages.size()*periodCount:periodCount; int readsPerScope=changesPurpose||combinedRanking?1:"ranking".equals(purpose)?fields.size()*stages.size():rawPurpose||defectPurpose?stages.size()*periodCount:periodCount;
@@ -248,7 +251,8 @@ final class AnalysisProtocolQueryPlan {
Map<String,Object> child=new LinkedHashMap<>(parent);child.remove("queryPlan");child.remove("comparisonMode"); Map<String,Object> child=new LinkedHashMap<>(parent);child.remove("queryPlan");child.remove("comparisonMode");
String purpose=string(node.get("purpose"));boolean ranking="ranking".equals(purpose),changes="changes".equals(purpose),defects="defects".equals(purpose),source="source".equals(purpose),records="records".equals(purpose),compare=Boolean.TRUE.equals(node.get("comparisonRequired")); String purpose=string(node.get("purpose"));boolean ranking="ranking".equals(purpose),changes="changes".equals(purpose),defects="defects".equals(purpose),source="source".equals(purpose),records="records".equals(purpose),compare=Boolean.TRUE.equals(node.get("comparisonRequired"));
child.put("kind",source?"source_verify":records?"details":singleObjectMetricRanking(parent,node,filters)?"metrics":ranking||defects||changes?"analysis":"metrics");child.put("filters",filters);child.put("fields",node.get("fields"));child.put("stages",node.get("stages")); child.put("kind",source?"source_verify":records?"details":singleObjectMetricRanking(parent,node,filters)?"metrics":ranking||defects||changes?"analysis":"metrics");child.put("filters",filters);child.put("fields",node.get("fields"));child.put("stages",node.get("stages"));
child.put("dimensions",ranking?List.of(node.get("dimension")):List.of());child.put("dimensionStages",ranking?Map.of(string(node.get("dimension")),node.get("stages")):Map.of()); boolean breakdown="metrics".equals(purpose)&&node.containsKey("dimension");
child.put("dimensions",ranking||breakdown?List.of(node.get("dimension")):List.of());child.put("dimensionStages",ranking||breakdown?Map.of(string(node.get("dimension")),node.get("stages")):Map.of());
child.remove("defectQuery");child.remove("changeQuery"); child.remove("defectQuery");child.remove("changeQuery");
if(changes){child.put("dimensions",List.of(node.get("dimension")));child.put("dimensionStages",Map.of(string(node.get("dimension")),node.get("stages")));child.put("changeQuery",Map.of("dimension",node.get("dimension"),"sort",node.get("sort"),"limit",node.get("limit")));} if(changes){child.put("dimensions",List.of(node.get("dimension")));child.put("dimensionStages",Map.of(string(node.get("dimension")),node.get("stages")));child.put("changeQuery",Map.of("dimension",node.get("dimension"),"sort",node.get("sort"),"limit",node.get("limit")));}
if(defects) { if(defects) {
@@ -64,7 +64,7 @@ final class AnalysisProtocolRankArtifact {
if(denominator.signum()<0)return Map.of(); if(denominator.signum()<0)return Map.of();
List<Map<String,Object>> rows=new ArrayList<>(); List<Map<String,Object>> rows=new ArrayList<>();
BigDecimal cumulative=BigDecimal.ZERO; BigDecimal cumulative=BigDecimal.ZERO;
for(int i=0;i<Math.min(10,positive.size());i++) { for(int i=0;i<Math.min(100,positive.size());i++) {
var row=new LinkedHashMap<>(positive.get(i));cumulative=cumulative.add((BigDecimal)row.get("weight")); var row=new LinkedHashMap<>(positive.get(i));cumulative=cumulative.add((BigDecimal)row.get("weight"));
row.put("rank",i+1);row.put("share",((BigDecimal)row.get("weight")).divide(denominator,java.math.MathContext.DECIMAL128)); row.put("rank",i+1);row.put("share",((BigDecimal)row.get("weight")).divide(denominator,java.math.MathContext.DECIMAL128));
row.put("cumulativeShare",cumulative.divide(denominator,java.math.MathContext.DECIMAL128));rows.add(row); row.put("cumulativeShare",cumulative.divide(denominator,java.math.MathContext.DECIMAL128));rows.add(row);
@@ -96,7 +96,7 @@ final class AnalysisProtocolRankArtifact {
} }
positive.sort((a,b)->{int weight=((BigDecimal)b.get("weight")).compareTo((BigDecimal)a.get("weight"));return weight!=0?weight:codePointCompare(a.get("key").toString(),b.get("key").toString());}); positive.sort((a,b)->{int weight=((BigDecimal)b.get("weight")).compareTo((BigDecimal)a.get("weight"));return weight!=0?weight:codePointCompare(a.get("key").toString(),b.get("key").toString());});
List<Map<String,Object>> rows=new ArrayList<>(); List<Map<String,Object>> rows=new ArrayList<>();
for(int i=0;i<Math.min(10,positive.size());i++){Map<String,Object> row=new LinkedHashMap<>(positive.get(i));row.put("rank",i+1);rows.add(row);} for(int i=0;i<Math.min(100,positive.size());i++){Map<String,Object> row=new LinkedHashMap<>(positive.get(i));row.put("rank",i+1);rows.add(row);}
Map<String,Object> rank=new LinkedHashMap<>();rank.put("stage",stage);rank.put("metric",field);rank.put("rows",rows);rank.put("totalRows",positive.size()); Map<String,Object> rank=new LinkedHashMap<>();rank.put("stage",stage);rank.put("metric",field);rank.put("rows",rows);rank.put("totalRows",positive.size());
rank.put("denominator",AnalysisProtocolService.map(AnalysisProtocolService.map(response.get("summary")).get(stage)).get(field)); rank.put("denominator",AnalysisProtocolService.map(AnalysisProtocolService.map(response.get("summary")).get(stage)).get(field));
rank.put("negativeGroups",negativeCount);rank.put("negativeWeight",negative);rank.put("orderPolicy","weight_desc_key_codepoint_asc");rank.put("completeRanking",true);rankings.add(rank); rank.put("negativeGroups",negativeCount);rank.put("negativeWeight",negative);rank.put("orderPolicy","weight_desc_key_codepoint_asc");rank.put("completeRanking",true);rankings.add(rank);
@@ -17,7 +17,7 @@ import org.springframework.web.server.ResponseStatusException;
import tools.jackson.databind.json.JsonMapper; import tools.jackson.databind.json.JsonMapper;
import tools.jackson.databind.DeserializationFeature; import tools.jackson.databind.DeserializationFeature;
/** Authoritative state and publication boundary for explicitly opted-in NAO sessions. */ /** Authoritative state and publication boundary for delegated DSH sessions. */
@Service @Service
public class AnalysisProtocolService { public class AnalysisProtocolService {
private static final Set<String> FILTERS = Set.of("department", "group", "product", "projectGroup", "extraFilters", "columnFilters", "exclusions", "includeUnassigned"); private static final Set<String> FILTERS = Set.of("department", "group", "product", "projectGroup", "extraFilters", "columnFilters", "exclusions", "includeUnassigned");
@@ -136,8 +136,8 @@ public class AnalysisProtocolService {
attemptId,sessionId,turnId,write(input)); attemptId,sessionId,turnId,write(input));
jdbc.update("UPDATE analysis_protocol_session SET current_attempt_id=? WHERE session_id=?",attemptId,sessionId); jdbc.update("UPDATE analysis_protocol_session SET current_attempt_id=? WHERE session_id=?",attemptId,sessionId);
Map<String,Object> plan=new LinkedHashMap<>(); Map<String,Object> plan=new LinkedHashMap<>();
plan.put("protocolVersion",1); plan.put("attemptId",attemptId); plan.put("nativeRuntime",true); plan.put("engine","NAO"); plan.put("protocolVersion",1); plan.put("attemptId",attemptId); plan.put("nativeRuntime",true); plan.put("engine",AnalysisRuntimeEngine.DSH);
plan.put("status","NATIVE_NAO_PENDING"); plan.put("steps",List.of()); plan.put("status",AnalysisRuntimeEngine.pendingStatus(AnalysisRuntimeEngine.DSH)); plan.put("steps",List.of());
plan.put("releaseId",inputContext(input).get("releaseId")); plan.put("frozenContext",inputContext(input)); plan.put("releaseId",inputContext(input).get("releaseId")); plan.put("frozenContext",inputContext(input));
Long floor=jdbc.queryForObject("SELECT COALESCE(MAX(id),0) FROM analysis_turn_event WHERE turn_id=?",Long.class,turnId); Long floor=jdbc.queryForObject("SELECT COALESCE(MAX(id),0) FROM analysis_turn_event WHERE turn_id=?",Long.class,turnId);
plan.put("nativeEventFloor",floor==null?0:floor); plan.put("nativeEventFloor",floor==null?0:floor);
@@ -185,7 +185,9 @@ public class AnalysisProtocolService {
if(count!=1)throw conflict("任务状态已变化"); if(count!=1)throw conflict("任务状态已变化");
jdbc.update("UPDATE analysis_protocol_attempt SET status='PLANNING' WHERE attempt_id=?",a.attemptId()); jdbc.update("UPDATE analysis_protocol_attempt SET status='PLANNING' WHERE attempt_id=?",a.attemptId());
inspection.updateAssociation(a,"RUNNING"); inspection.updateAssociation(a,"RUNNING");
event(a.attemptId(),"progress",Map.of("turnId",turnId,"status","RUNNING","message","正在理解并验证当前任务…")); var startedAt = jdbc.queryForObject("SELECT started_at FROM analysis_turn WHERE id=?", java.sql.Timestamp.class, turnId);
event(a.attemptId(),"progress",Map.of("turnId",turnId,"status","RUNNING","message","正在理解并验证当前任务…",
"startedAt", startedAt.toInstant().toString()));
return require(a.attemptId(),actor,false); return require(a.attemptId(),actor,false);
} }
@@ -326,8 +328,16 @@ public class AnalysisProtocolService {
metadata.put("sourceStatus",input.get("sourceStatus"));metadata.put("queryRefs",checked.get("queryRefs"));metadata.put("missingEvidence",checked.get("missingEvidence")); metadata.put("sourceStatus",input.get("sourceStatus"));metadata.put("queryRefs",checked.get("queryRefs"));metadata.put("missingEvidence",checked.get("missingEvidence"));
var overlay=new LinkedHashMap<String,Object>();for(String key:List.of("visualSchemaVersion","visualState","visuals"))overlay.put(key,checked.get(key));overlay.put("visualRecovery",metadata); var overlay=new LinkedHashMap<String,Object>();for(String key:List.of("visualSchemaVersion","visualState","visuals"))overlay.put(key,checked.get(key));overlay.put("visualRecovery",metadata);
boolean originalComplete="COMPLETED".equals(input.get("sourceStatus")); boolean originalComplete="COMPLETED".equals(input.get("sourceStatus"));
boolean factsComplete=Boolean.TRUE.equals(map(map(input.get("sourceAnswer")).get("factCoverage")).get("complete"))&&Boolean.TRUE.equals(input.get("factsStillCovered")); var sourceAnswer=map(input.get("sourceAnswer"));var quality=new LinkedHashMap<>(map(sourceAnswer.get("answerQuality")));
if(!originalComplete&&factsComplete&&Boolean.TRUE.equals(checked.get("chartsComplete"))) { boolean chartsComplete=Boolean.TRUE.equals(checked.get("chartsComplete"));
if(!quality.isEmpty()){quality.put("chartsComplete",chartsComplete);overlay.put("answerQuality",quality);}
boolean factsComplete=Boolean.TRUE.equals(map(sourceAnswer.get("factCoverage")).get("complete"))&&Boolean.TRUE.equals(input.get("factsStillCovered"));
boolean narrativeComplete=Boolean.TRUE.equals(quality.get("narrativeComplete"))&&Boolean.TRUE.equals(quality.get("dataComplete"))
&&!Boolean.FALSE.equals(sourceAnswer.get("nativeStopSucceeded"))
&&!Boolean.FALSE.equals(quality.get("citationComplete"))&&list(quality.get("unresolvedEvidenceRefs")).isEmpty()
&&!Objects.toString(sourceAnswer.get("answer"),"").isBlank();
if(!originalComplete&&"PARTIAL".equals(input.get("sourceStatus"))&&factsComplete&&narrativeComplete&&chartsComplete) {
metadata.put("priorMissingEvidence",list(quality.get("missingEvidence")));quality.put("missingEvidence",List.of());
overlay.put("status","COMPLETED");overlay.put("originalStatus",input.get("sourceStatus")); overlay.put("status","COMPLETED");overlay.put("originalStatus",input.get("sourceStatus"));
overlay.put("completion",Map.of("completed",true,"reason","visual_recovery_complete"));overlay.put("stopReason","visual_recovery_complete"); overlay.put("completion",Map.of("completed",true,"reason","visual_recovery_complete"));overlay.put("stopReason","visual_recovery_complete");
} }
@@ -421,7 +431,7 @@ public class AnalysisProtocolService {
jdbc.update("UPDATE analysis_protocol_session SET status='FAILED',version=version+1 WHERE session_id=?",a.sessionId()); jdbc.update("UPDATE analysis_protocol_session SET status='FAILED',version=version+1 WHERE session_id=?",a.sessionId());
jdbc.update("UPDATE analysis_protocol_attempt SET status='FAILED' WHERE attempt_id=?",id); jdbc.update("UPDATE analysis_protocol_attempt SET status='FAILED' WHERE attempt_id=?",id);
boolean nativeConversation=autonomousSession(a.sessionId())&&!a.input().containsKey("controlledInspection"); boolean nativeConversation=autonomousSession(a.sessionId())&&!a.input().containsKey("controlledInspection");
Map<String,Object> answer=Map.of("answer",nativeConversation?"":reason,"results",List.of(),"releaseId",a.releaseId(),"engine",a.input().containsKey("controlledInspection")?"CONTROLLED_INSPECTION_PROTOCOL":"NAO","completion",Map.of("completed",false,"reason",reason)); Map<String,Object> answer=Map.of("answer",nativeConversation?"":reason,"results",List.of(),"releaseId",a.releaseId(),"engine",frozenTurnEngine(a),"completion",Map.of("completed",false,"reason",reason));
jdbc.update("UPDATE analysis_turn SET status='FAILED',answer_json=CAST(? AS JSON),stop_reason=?,finished_at=CURRENT_TIMESTAMP(6) WHERE id=?",write(answer),reason,a.turnId()); jdbc.update("UPDATE analysis_turn SET status='FAILED',answer_json=CAST(? AS JSON),stop_reason=?,finished_at=CURRENT_TIMESTAMP(6) WHERE id=?",write(answer),reason,a.turnId());
inspection.updateAssociation(a,"FAILED"); inspection.updateAssociation(a,"FAILED");
event(id,"failed",Map.of("turnId",a.turnId(),"status","FAILED","result",answer,"stopReason",reason)); event(id,"failed",Map.of("turnId",a.turnId(),"status","FAILED","result",answer,"stopReason",reason));
@@ -440,7 +450,7 @@ public class AnalysisProtocolService {
boolean clarification=Boolean.TRUE.equals(task.get("needsClarification")); boolean clarification=Boolean.TRUE.equals(task.get("needsClarification"));
var state=store.state(a.sessionId()); var state=store.state(a.sessionId());
Map<String,Object> retry=map(a.input().get("retryTask")); Map<String,Object> retry=map(a.input().get("retryTask"));
if(!retry.isEmpty()) { if(!retry.isEmpty()&&!Boolean.TRUE.equals(task.get("agentDriven"))) {
if(!Objects.equals(canonical(retry),canonical(task)))throw conflict("重试不能更改已确认任务,请新建分析轮次"); if(!Objects.equals(canonical(retry),canonical(task)))throw conflict("重试不能更改已确认任务,请新建分析轮次");
} else { } else {
state=store.appendRevision(a.sessionId(),state.version(),clarification?AnalysisExecutionProtocolStore.Disposition.CLARIFICATION:AnalysisExecutionProtocolStore.Disposition.CONFIRMED,write(task)); state=store.appendRevision(a.sessionId(),state.version(),clarification?AnalysisExecutionProtocolStore.Disposition.CLARIFICATION:AnalysisExecutionProtocolStore.Disposition.CONFIRMED,write(task));
@@ -574,7 +584,7 @@ public class AnalysisProtocolService {
int returned=list(data.get("rows")).size(); int returned=list(data.get("rows")).size();
coverage.put("rowCount",map(data.get("context")).getOrDefault("rowCount",returned)); coverage.put("rowCount",map(data.get("context")).getOrDefault("rowCount",returned));
coverage.put("returnedRows",returned); coverage.put("returnedRows",returned);
if(list(child.get("dimensions")).contains(params.get("view"))) { if(!"metrics".equals(plannedScope.get("purpose"))&&list(child.get("dimensions")).contains(params.get("view"))) {
boolean complete=AnalysisProtocolRankingCoverage.complete(data,child,params.get("view")); boolean complete=AnalysisProtocolRankingCoverage.complete(data,child,params.get("view"));
if(declared.containsKey("queryPlan")||AnalysisProtocolExploration.nativeAgent(declared)) { if(declared.containsKey("queryPlan")||AnalysisProtocolExploration.nativeAgent(declared)) {
Map<String,Object> summaryTask=new LinkedHashMap<>(child);summaryTask.put("trendRequired",false); Map<String,Object> summaryTask=new LinkedHashMap<>(child);summaryTask.put("trendRequired",false);
@@ -599,6 +609,7 @@ public class AnalysisProtocolService {
@Transactional @Transactional
public void progress(String attemptId,Map<String,Object> progress,AppUserPrincipal actor) { public void progress(String attemptId,Map<String,Object> progress,AppUserPrincipal actor) {
if (progress.containsKey("dshDraft")) throw conflict("临时输出不能写入分析记录");
lock(sessionForAttempt(attemptId)); lock(sessionForAttempt(attemptId));
Attempt a=require(attemptId,actor,false); Attempt a=require(attemptId,actor,false);
if(!"RUNNING".equals(a.turnStatus()))throw conflict("本轮已结束"); if(!"RUNNING".equals(a.turnStatus()))throw conflict("本轮已结束");
@@ -609,6 +620,8 @@ public class AnalysisProtocolService {
if(!delta.isEmpty())event(attemptId,"progress",Map.of("turnId",a.turnId(),"status","RUNNING","thinkingDelta",delta)); if(!delta.isEmpty())event(attemptId,"progress",Map.of("turnId",a.turnId(),"status","RUNNING","thinkingDelta",delta));
} else { } else {
plan.put("progressMessage",Objects.toString(progress.get("message"),"正在分析…")); plan.put("progressMessage",Objects.toString(progress.get("message"),"正在分析…"));
if(progress.containsKey("dshActivity"))plan.put("dshActivity",progress.get("dshActivity"));
else plan.remove("dshActivity");
if(progress.containsKey("steps"))plan.put("steps",progress.get("steps")); if(progress.containsKey("steps"))plan.put("steps",progress.get("steps"));
if(progress.containsKey("timeline"))plan.put("timeline",progress.get("timeline")); if(progress.containsKey("timeline"))plan.put("timeline",progress.get("timeline"));
jdbc.update("UPDATE analysis_turn SET plan_json=CAST(? AS JSON),steps_json=CAST(? AS JSON) WHERE id=?",write(plan),write(plan.getOrDefault("steps",List.of())),a.turnId()); jdbc.update("UPDATE analysis_turn SET plan_json=CAST(? AS JSON),steps_json=CAST(? AS JSON) WHERE id=?",write(plan),write(plan.getOrDefault("steps",List.of())),a.turnId());
@@ -625,6 +638,11 @@ public class AnalysisProtocolService {
if(!actor.getId().toString().equals(a.userId()) || !a.scopeHash().equals(AuthorizationFingerprint.of(actor,governance)))throw denied("授权范围已变化"); if(!actor.getId().toString().equals(a.userId()) || !a.scopeHash().equals(AuthorizationFingerprint.of(actor,governance)))throw denied("授权范围已变化");
governance.requireBoundAction("QUERY","query.detail");if(!a.input().containsKey("controlledInspection"))governance.requireBoundAction("MODEL","model.intent.parse"); governance.requireBoundAction("QUERY","query.detail");if(!a.input().containsKey("controlledInspection"))governance.requireBoundAction("MODEL","model.intent.parse");
options.options(a.releaseId(),actor); options.options(a.releaseId(),actor);
String frozenEngine=frozenTurnEngine(a);
Map<String,Object> originalOutput = new LinkedHashMap<>(details);
originalOutput.putAll(answer);
String original = modelAnswer(details, answer);
if (!original.isBlank()) originalOutput.put("modelAnswer", original);
answer=new LinkedHashMap<>(answer);for(String reserved:List.of("memberList","recordList","protocolQueryIds","executionScopes","executionCoverage","factCoverage","attemptProtocolQueryIds"))answer.remove(reserved); answer=new LinkedHashMap<>(answer);for(String reserved:List.of("memberList","recordList","protocolQueryIds","executionScopes","executionCoverage","factCoverage","attemptProtocolQueryIds"))answer.remove(reserved);
String status=requestedStatus; String status=requestedStatus;
if(!Set.of("COMPLETED","PARTIAL","FAILED","CANCELLED","NEEDS_CLARIFICATION").contains(status))status="FAILED"; if(!Set.of("COMPLETED","PARTIAL","FAILED","CANCELLED","NEEDS_CLARIFICATION").contains(status))status="FAILED";
@@ -632,24 +650,33 @@ public class AnalysisProtocolService {
binding.current(a.sessionId(),a.turnId(),a.userId(),a.revision(),a.executionId()); binding.current(a.sessionId(),a.turnId(),a.userId(),a.revision(),a.executionId());
Map<String,Object> task=read(store.revision(a.sessionId(),a.revision()).taskJson()); Map<String,Object> task=read(store.revision(a.sessionId(),a.revision()).taskJson());
if(!Objects.equals(canonical(task),canonical(details.get("agentTask")))) throw denied("最终答复没有绑定当前任务"); if(!Objects.equals(canonical(task),canonical(details.get("agentTask")))) throw denied("最终答复没有绑定当前任务");
if(Set.of("COMPLETED","PARTIAL").contains(status)&&AnalysisDirectConversation.applies(task)) { boolean directDsh = !a.input().containsKey("controlledInspection")
&& AnalysisRuntimeEngine.DSH.equals(frozenEngine)
&& "dsh_direct_answer".equals(details.get("completionMethod"));
if(Set.of("COMPLETED","PARTIAL").contains(status)&&directDsh) {
// The worker publishes the original DSH body. Keep real query
// IDs for history, without post-answer content/coverage review.
answer.put("protocolQueryIds",jdbc.query(
"SELECT query_id FROM analysis_protocol_query WHERE attempt_id=? ORDER BY created_at,query_id",
(rs,n)->rs.getString(1),a.attemptId()));
for(String key:List.of("answerQuality","factCoverage","executionCoverage","scopeCoverage","explorationWarnings"))
answer.remove(key);
answer.put("answerValidation",Map.of("mode","disabled","originalPreserved",true));
if("report".equals(task.get("kind")))
for(String key:List.of("historySources","historySections","historyCoverage","historyEvidence","historyVisualSources"))
if(details.containsKey(key))answer.put(key,details.get(key));
} else if(Set.of("COMPLETED","PARTIAL").contains(status)&&AnalysisDirectConversation.applies(task)) {
answer=AnalysisDirectConversation.answer(task,a.releaseId(),taskContext(a.releaseId(),task)); answer=AnalysisDirectConversation.answer(task,a.releaseId(),taskContext(a.releaseId(),task));
status="COMPLETED"; status="COMPLETED";
} else if(Set.of("COMPLETED","PARTIAL").contains(status)&&!"report".equals(task.get("kind")) } else if(Set.of("COMPLETED","PARTIAL").contains(status)&&!"report".equals(task.get("kind"))
&&!hasProtocolEvidence(details)) { &&!hasProtocolEvidence(details)) {
status="FAILED"; status="FAILED";
String original=AnalysisProtocolExploration.nativeAgent(task)?Objects.toString(answer.get("answer"),""):null;
answer.clear(); answer.clear();
answer.put("answer",original!=null?original:noProofFailureReason(details)); answer.put("answer",noProofFailureReason(details));
answer.put("results",List.of()); answer.put("results",List.of());
answer.put("releaseId",a.releaseId()); answer.put("releaseId",a.releaseId());
answer.put("engine","NAO"); answer.put("engine",frozenEngine);
answer.put("stopReason","formal_query_evidence_missing"); answer.put("stopReason","formal_query_evidence_missing");
if(original!=null) {
answer.put("executionMode","native_agent");
answer.put("answerQuality",Map.of("version",1,"dataComplete",false,"narrativeFactBinding","not_assessed",
"missingEvidence",List.of(noProofFailureReason(details))));
}
} else if(Set.of("COMPLETED","PARTIAL").contains(status)) { } else if(Set.of("COMPLETED","PARTIAL").contains(status)) {
governance.requireBoundAction("QUERY","query.aggregate"); governance.requireBoundAction("QUERY","query.aggregate");
if(AnalysisProtocolInspection.KIND.equals(task.get("kind"))) { if(AnalysisProtocolInspection.KIND.equals(task.get("kind"))) {
@@ -727,11 +754,30 @@ public class AnalysisProtocolService {
if(!Boolean.TRUE.equals(map(details.get("evidenceCompletion")).get("serverCoverageSufficient")))status="PARTIAL"; if(!Boolean.TRUE.equals(map(details.get("evidenceCompletion")).get("serverCoverageSufficient")))status="PARTIAL";
if(!task.containsKey("queryPlan")&&!AnalysisProtocolExploration.nativeAgent(task)&&Boolean.TRUE.equals(task.get("trendRequired"))&&!"none".equals(task.get("visualPreference"))&&!"ready".equals(answer.get("visualState")))status="PARTIAL"; if(!task.containsKey("queryPlan")&&!AnalysisProtocolExploration.nativeAgent(task)&&Boolean.TRUE.equals(task.get("trendRequired"))&&!"none".equals(task.get("visualPreference"))&&!"ready".equals(answer.get("visualState")))status="PARTIAL";
answer.put("factCoverage",Map.of("version",1,"complete",factsComplete)); answer.put("factCoverage",Map.of("version",1,"complete",factsComplete));
// The exportable artifact must be the exact evidence-checked body
// that survived server-side redaction and coverage handling.
retainCheckedDeliverable(answer);
} }
} }
} else if("COMPLETED".equals(status) || "PARTIAL".equals(status)) status="FAILED"; } else if("COMPLETED".equals(status) || "PARTIAL".equals(status)) {
status="FAILED";
answer.clear();
answer.put("answer",noProofFailureReason(details));
answer.put("results",List.of());
answer.put("releaseId",a.releaseId());
answer.put("stopReason","formal_query_evidence_missing");
}
if("CLARIFICATION".equals(a.status()) && !"NEEDS_CLARIFICATION".equals(status))status="FAILED"; if("CLARIFICATION".equals(a.status()) && !"NEEDS_CLARIFICATION".equals(status))status="FAILED";
answer=new LinkedHashMap<>(answer);answer.put("completion",Map.of("completed","COMPLETED".equals(status),"reason",Objects.toString(answer.get("stopReason"),""))); if(!Set.of("COMPLETED","PARTIAL").contains(status))
for(String key:List.of("reportMarkdown","reportHtml","reportTitle"))answer.remove(key);
boolean modelAuthored = !a.input().containsKey("controlledInspection");
if (modelAuthored) preserveModelOutput(answer, originalOutput);
if (modelAuthored && originalOutput.containsKey("modelAnswer") && original.isBlank()) {
answer.put("answer", "");
answer.put("modelAnswer", "");
for (String key : List.of("reportMarkdown","reportHtml","reportTitle")) answer.remove(key);
}
answer=new LinkedHashMap<>(answer);answer.put("engine",frozenEngine);answer.put("completion",Map.of("completed","COMPLETED".equals(status),"reason",Objects.toString(answer.get("stopReason"),"")));
answer.put("protocol",identity(a)); answer.put("protocol",identity(a));
if(a.executionId()!=null) { if(a.executionId()!=null) {
var outcome="COMPLETED".equals(status)?AnalysisExecutionProtocolStore.Outcome.COMPLETED: var outcome="COMPLETED".equals(status)?AnalysisExecutionProtocolStore.Outcome.COMPLETED:
@@ -740,6 +786,7 @@ public class AnalysisProtocolService {
} }
jdbc.update("UPDATE analysis_protocol_attempt SET status=? WHERE attempt_id=?",status,attemptId); jdbc.update("UPDATE analysis_protocol_attempt SET status=? WHERE attempt_id=?",status,attemptId);
Map<String,Object> plan=read(jdbc.queryForObject("SELECT plan_json FROM analysis_turn WHERE id=?",String.class,a.turnId())); Map<String,Object> plan=read(jdbc.queryForObject("SELECT plan_json FROM analysis_turn WHERE id=?",String.class,a.turnId()));
plan.remove("dshActivity");
if(answer.get("context") instanceof Map<?,?>)plan.put("context",answer.get("context")); if(answer.get("context") instanceof Map<?,?>)plan.put("context",answer.get("context"));
List<Map<String,Object>> steps=AnalysisTurnService.nativeSteps(details.get("toolCalls"));plan.put("steps",steps); List<Map<String,Object>> steps=AnalysisTurnService.nativeSteps(details.get("toolCalls"));plan.put("steps",steps);
jdbc.update("UPDATE analysis_turn SET status=?,answer_json=CAST(? AS JSON),plan_json=CAST(? AS JSON),steps_json=CAST(? AS JSON),stop_reason=?,finished_at=CURRENT_TIMESTAMP(6) WHERE id=? AND status='RUNNING'", jdbc.update("UPDATE analysis_turn SET status=?,answer_json=CAST(? AS JSON),plan_json=CAST(? AS JSON),steps_json=CAST(? AS JSON),stop_reason=?,finished_at=CURRENT_TIMESTAMP(6) WHERE id=? AND status='RUNNING'",
@@ -748,18 +795,104 @@ public class AnalysisProtocolService {
event(attemptId,status.toLowerCase(Locale.ROOT),Map.of("turnId",a.turnId(),"status",status,"result",answer)); event(attemptId,status.toLowerCase(Locale.ROOT),Map.of("turnId",a.turnId(),"status",status,"result",answer));
} }
static void retainCheckedDeliverable(Map<String,Object> answer) {
answer.remove("reportHtml");
if (!(answer.get("reportMarkdown") instanceof String markdown)
|| markdown.isBlank() || !markdown.equals(answer.get("answer"))) {
answer.remove("reportMarkdown");
answer.remove("reportTitle");
}
}
@SafeVarargs
static String modelAnswer(Map<String,Object>... sources) {
boolean explicit = false;
for (String field : List.of("modelAnswer", "answer")) {
if (field.equals("answer") && explicit) return "";
for (Map<String,Object> source : sources) {
if (source.get(field) instanceof String original) {
if (field.equals("modelAnswer")) explicit = true;
if (!original.isBlank()) return DshAnalysisRunner.publicText(original);
}
}
}
return "";
}
static void preserveModelOutput(Map<String,Object> answer, Map<String,Object> output) {
String original = modelAnswer(output);
if (original.isBlank()) return;
answer.put("modelAnswer", original);
answer.put("answer", original);
if (output.get("answerOrigin") instanceof String origin && origin.matches("[a-z_]{1,80}")) answer.put("answerOrigin",origin);
if (output.get("answerValidation") instanceof Map<?,?> validation) {
var safe = new LinkedHashMap<String,Object>();
for (String field : List.of("mode", "status", "stopReason", "originalPreserved")) {
Object value = validation.get(field);
if (value instanceof String text) safe.put(field,DshAnalysisRunner.publicText(text.substring(0,Math.min(text.length(),1000))));
else if (value instanceof Boolean) safe.put(field,value);
}
answer.put("answerValidation",safe);
}
if (output.get("reportMarkdown") instanceof String markdown && original.equals(DshAnalysisRunner.publicText(markdown))) {
answer.put("reportMarkdown", original);
answer.remove("reportHtml");
if (output.get("reportTitle") instanceof String title && !title.isBlank() && title.length() <= 160)
answer.put("reportTitle", DshAnalysisRunner.publicText(title));
} else if (answer.get("reportMarkdown") instanceof String markdown && !original.equals(markdown)) {
for (String field : List.of("reportMarkdown", "reportHtml", "reportTitle")) answer.remove(field);
}
}
/** Recovery keeps the same authorized attempt's model prose after a stop or a failed validation. */
@Transactional
public void preserveStoppedModelOutput(String id, Map<String,Object> output, AppUserPrincipal actor) {
if (modelAnswer(output).isBlank()) return;
lock(sessionForAttempt(id));
Attempt a = attempt(id);
if (!a.attemptId().equals(a.currentAttempt()) || !Set.of("FAILED", "CANCELLED").contains(a.turnStatus())) return;
if (!actor.getId().toString().equals(a.userId()) || !a.scopeHash().equals(AuthorizationFingerprint.of(actor,governance)))
throw denied("授权范围已变化");
governance.requireDetail(actor);
governance.requireBoundAction("QUERY", "query.detail");
options.options(a.releaseId(),actor);
Map<String,Object> saved = new LinkedHashMap<>(read(jdbc.queryForObject("SELECT answer_json FROM analysis_turn WHERE id=?",String.class,a.turnId())));
preserveModelOutput(saved, output);
saved.put("engine",frozenTurnEngine(a));
saved.put("releaseId",a.releaseId());
saved.put("protocol",identity(a));
saved.putIfAbsent("completion",Map.of("completed",false,"reason",a.turnStatus()));
jdbc.update("UPDATE analysis_turn SET answer_json=CAST(? AS JSON) WHERE id=? AND status IN ('FAILED','CANCELLED')",write(saved),a.turnId());
event(id,a.turnStatus().toLowerCase(Locale.ROOT),Map.of("turnId",a.turnId(),"status",a.turnStatus(),"result",saved));
}
/** Final result labels always come from the immutable server plan, never the worker payload. */
private String frozenTurnEngine(Attempt attempt) {
Map<String,Object> plan=read(jdbc.queryForObject("SELECT plan_json FROM analysis_turn WHERE id=?",String.class,attempt.turnId()));
if(attempt.input().containsKey("controlledInspection")||"CONTROLLED_INSPECTION_PROTOCOL".equals(plan.get("engine")))
return "CONTROLLED_INSPECTION_PROTOCOL";
return AnalysisRuntimeEngine.fromPlan(plan);
}
@Transactional @Transactional
public Map<String,Object> cancel(String turnId,AppUserPrincipal actor,String reason) { public Map<String,Object> cancel(String turnId,AppUserPrincipal actor,String reason) {
return cancel(turnId,actor,reason,Map.of());
}
@Transactional
public Map<String,Object> cancel(String turnId,AppUserPrincipal actor,String reason,Map<String,Object> output) {
String id=currentAttemptIdForTurn(turnId);lock(sessionForAttempt(id)); String id=currentAttemptIdForTurn(turnId);lock(sessionForAttempt(id));
Attempt a=attempt(id); Attempt a=attempt(id);
if(!a.userId().equals(actor.getId().toString()))throw denied("会话不属于当前用户"); if(!a.userId().equals(actor.getId().toString()))throw denied("会话不属于当前用户");
if(!a.scopeHash().equals(AuthorizationFingerprint.of(actor,governance)))throw denied("授权范围已变化");
if(!Set.of("QUEUED","PLANNING","RUNNING","CLARIFICATION").contains(a.status()))return Map.of("turnId",turnId,"status",a.turnStatus(),"cancelled",false); if(!Set.of("QUEUED","PLANNING","RUNNING","CLARIFICATION").contains(a.status()))return Map.of("turnId",turnId,"status",a.turnStatus(),"cancelled",false);
var state=store.state(a.sessionId()); var state=store.state(a.sessionId());
store.cancel(a.sessionId(),state.version()); store.cancel(a.sessionId(),state.version());
jdbc.update("UPDATE analysis_protocol_attempt SET status='CANCELLED' WHERE attempt_id=?",a.attemptId()); jdbc.update("UPDATE analysis_protocol_attempt SET status='CANCELLED' WHERE attempt_id=?",a.attemptId());
jdbc.update("UPDATE analysis_turn SET status='CANCELLED',cancel_requested=TRUE,stop_reason=?,finished_at=CURRENT_TIMESTAMP(6) WHERE id=?",reason,turnId); jdbc.update("UPDATE analysis_turn SET status='CANCELLED',cancel_requested=TRUE,stop_reason=?,finished_at=CURRENT_TIMESTAMP(6) WHERE id=?",reason,turnId);
inspection.updateAssociation(a,"CANCELLED"); inspection.updateAssociation(a,"CANCELLED");
event(a.attemptId(),"cancelled",Map.of("turnId",turnId,"status","CANCELLED","stopReason",reason)); if (!modelAnswer(output).isBlank()) preserveStoppedModelOutput(id,output,actor);
else event(a.attemptId(),"cancelled",Map.of("turnId",turnId,"status","CANCELLED","stopReason",reason));
return Map.of("turnId",turnId,"status","CANCELLED","cancelRequested",true); return Map.of("turnId",turnId,"status","CANCELLED","cancelRequested",true);
} }
@@ -856,7 +989,11 @@ public class AnalysisProtocolService {
lock(sessionForAttempt(attemptId));var a=require(attemptId,actor,true); lock(sessionForAttempt(attemptId));var a=require(attemptId,actor,true);
binding.current(a.sessionId(),a.turnId(),a.userId(),a.revision(),a.executionId());var root=task(a); binding.current(a.sessionId(),a.turnId(),a.userId(),a.revision(),a.executionId());var root=task(a);
if(!autonomousSession(a.sessionId())||!AnalysisProtocolExploration.nativeAgent(root))throw denied("当前任务没有授权自主探索"); if(!autonomousSession(a.sessionId())||!AnalysisProtocolExploration.nativeAgent(root))throw denied("当前任务没有授权自主探索");
if(request==null||!Set.of("requestKey","node","parentQueryIds").containsAll(request.keySet())||!(request.get("requestKey") instanceof String key)||key.isBlank()||key.length()>128||!(request.get("node") instanceof Map<?,?>))throw conflict("自主步骤格式无效"); if(request==null||!Set.of("requestKey","node","parentQueryIds","frame").containsAll(request.keySet())||!(request.get("requestKey") instanceof String key)||key.isBlank()||key.length()>128||!(request.get("node") instanceof Map<?,?>))throw conflict("自主步骤格式无效");
if(request.containsKey("frame")) {
try {root=AnalysisProtocolExploration.withFrame(root,map(request.get("frame")),planCatalog(a,root,actor));}
catch(IllegalArgumentException invalid){throw conflict(invalid.getMessage());}
}
var parents=list(request.getOrDefault("parentQueryIds",List.of())); var parents=list(request.getOrDefault("parentQueryIds",List.of()));
if(!(request.getOrDefault("parentQueryIds",List.of()) instanceof List<?>)||parents.size()>8||new HashSet<>(parents).size()!=parents.size()||parents.stream().anyMatch(p->!(p instanceof String)))throw conflict("父查询证明格式无效"); if(!(request.getOrDefault("parentQueryIds",List.of()) instanceof List<?>)||parents.size()>8||new HashSet<>(parents).size()!=parents.size()||parents.stream().anyMatch(p->!(p instanceof String)))throw conflict("父查询证明格式无效");
String digest=hash(write(canonical(request))); String digest=hash(write(canonical(request)));
@@ -1000,7 +1137,7 @@ public class AnalysisProtocolService {
if("GET /api/metrics".equals(target)) { if("GET /api/metrics".equals(target)) {
requireHistoricalMetricContext(release,child,parameters,response); requireHistoricalMetricContext(release,child,parameters,response);
coverage.putAll(AnalysisProtocolMetricCoverage.inspect(response,child,queriedPeriod(parameters))); coverage.putAll(AnalysisProtocolMetricCoverage.inspect(response,child,queriedPeriod(parameters)));
if(list(child.get("dimensions")).contains(parameters.get("view"))) { if(!"metrics".equals(scope.get("purpose"))&&list(child.get("dimensions")).contains(parameters.get("view"))) {
boolean complete=AnalysisProtocolRankingCoverage.complete(response,child,parameters.get("view"));coverage.put("weightsComplete",complete); boolean complete=AnalysisProtocolRankingCoverage.complete(response,child,parameters.get("view"));coverage.put("weightsComplete",complete);
if(complete) {artifact=new LinkedHashMap<>(AnalysisProtocolRankArtifact.build(response,child,parameters.get("view"),release));artifact.put("displayLimit",scope.getOrDefault("limit",10));} if(complete) {artifact=new LinkedHashMap<>(AnalysisProtocolRankArtifact.build(response,child,parameters.get("view"),release));artifact.put("displayLimit",scope.getOrDefault("limit",10));}
} }
@@ -1049,7 +1186,7 @@ public class AnalysisProtocolService {
return new AnalysisProtocolQueryPlan(json).resolve(task,a.executionId(),a.releaseId(),planCatalog(a,task,actor),records,availableRankings(a,actor)); return new AnalysisProtocolQueryPlan(json).resolve(task,a.executionId(),a.releaseId(),planCatalog(a,task,actor),records,availableRankings(a,actor));
} }
private Map<String,Object> planCatalog(Attempt a,Map<String,Object> task,AppUserPrincipal actor) { private Map<String,Object> planCatalog(Attempt a,Map<String,Object> task,AppUserPrincipal actor) {
Map<String,Object> catalog=new LinkedHashMap<>(options.options(a.releaseId(),actor)); Map<String,Object> catalog=new LinkedHashMap<>(AnalysisAgentCatalog.active(options.options(a.releaseId(),actor)));
if(Boolean.TRUE.equals(task.get("verifySource"))||Boolean.TRUE.equals(task.get("fullDetails"))) { if(Boolean.TRUE.equals(task.get("verifySource"))||Boolean.TRUE.equals(task.get("fullDetails"))) {
governance.requireEvidence(actor); governance.requireEvidence(actor);
if(sourceColumns==null)throw conflict("源表调查目录当前不可用"); if(sourceColumns==null)throw conflict("源表调查目录当前不可用");
@@ -1092,7 +1229,7 @@ public class AnalysisProtocolService {
.map(body->Map.of("body",write(canonical(body)))).toList(); .map(body->Map.of("body",write(canonical(body)))).toList();
Map<String,Object> child=map(scope.get("task")); Map<String,Object> child=map(scope.get("task"));
List<Map<String,String>> requests=new ArrayList<>();Map<String,String> current=new LinkedHashMap<>(metricParameters(release,child)); List<Map<String,String>> requests=new ArrayList<>();Map<String,String> current=new LinkedHashMap<>(metricParameters(release,child));
if("ranking".equals(scope.get("purpose")))current.put("view",list(child.get("dimensions")).getFirst().toString()); if(("ranking".equals(scope.get("purpose"))||"metrics".equals(scope.get("purpose")))&&!list(child.get("dimensions")).isEmpty())current.put("view",list(child.get("dimensions")).getFirst().toString());
requests.add(current); requests.add(current);
if(Boolean.TRUE.equals(child.get("comparisonRequired"))) { if(Boolean.TRUE.equals(child.get("comparisonRequired"))) {
Map<String,Object> previous=new LinkedHashMap<>(child);previous.put("period",child.get("comparisonPeriod"));requests.add(metricParameters(release,previous)); Map<String,Object> previous=new LinkedHashMap<>(child);previous.put("period",child.get("comparisonPeriod"));requests.add(metricParameters(release,previous));
@@ -1624,7 +1761,7 @@ public class AnalysisProtocolService {
if(!rows.isEmpty()&&!rows.getFirst().get("scope").equals(AuthorizationFingerprint.of(actor,governance)))throw denied("历史结果的授权范围已变化"); if(!rows.isEmpty()&&!rows.getFirst().get("scope").equals(AuthorizationFingerprint.of(actor,governance)))throw denied("历史结果的授权范围已变化");
governance.requireBoundAction("QUERY","query.aggregate"); governance.requireBoundAction("QUERY","query.aggregate");
String release=jdbc.queryForObject("SELECT release_id FROM analysis_session WHERE id=? AND user_id=?",String.class,sessionId,actor.getId().toString()); String release=jdbc.queryForObject("SELECT release_id FROM analysis_session WHERE id=? AND user_id=?",String.class,sessionId,actor.getId().toString());
options.options(release,actor); options.requirePublishedRelease(release);
} }
/** Only called inside the existing, explicitly requested session-delete transaction. */ /** Only called inside the existing, explicitly requested session-delete transaction. */
@Transactional @Transactional
@@ -1644,6 +1781,11 @@ public class AnalysisProtocolService {
Map<String,Object> t=new LinkedHashMap<>(proposed==null?Map.of():proposed); Map<String,Object> t=new LinkedHashMap<>(proposed==null?Map.of():proposed);
if(!Objects.equals(t.get("version"),1) || !(t.get("goal") instanceof String goal) || goal.isBlank() || goal.length()>1600 if(!Objects.equals(t.get("version"),1) || !(t.get("goal") instanceof String goal) || goal.isBlank() || goal.length()>1600
|| !(t.get("needsClarification") instanceof Boolean) || !Objects.equals(t.get("question"),question(a.turnId())))throw conflict("任务提案格式或所属轮次不正确"); || !(t.get("needsClarification") instanceof Boolean) || !Objects.equals(t.get("question"),question(a.turnId())))throw conflict("任务提案格式或所属轮次不正确");
if(Boolean.TRUE.equals(t.get("agentDriven"))) {
if(!AnalysisProtocolExploration.nativeAgent(t)||!autonomousSession(a.sessionId()))throw denied("当前会话没有自主执行权限");
options.options(a.releaseId(),actor);
return AnalysisProtocolExploration.start(t);
}
if("report".equals(t.get("kind")))return history.freeze(a,t,actor,autonomousSession(a.sessionId())); if("report".equals(t.get("kind")))return history.freeze(a,t,actor,autonomousSession(a.sessionId()));
Map<String,Object> available=options.options(a.releaseId(),actor); Map<String,Object> available=options.options(a.releaseId(),actor);
if(!(t.get("filters") instanceof Map<?,?>))throw conflict("任务必须完整声明筛选对象"); if(!(t.get("filters") instanceof Map<?,?>))throw conflict("任务必须完整声明筛选对象");
@@ -1897,6 +2039,7 @@ public class AnalysisProtocolService {
catch(NumberFormatException e){throw denied("行数限制无效");} catch(NumberFormatException e){throw denied("行数限制无效");}
} }
if(out.containsKey("rowOffset")&&!Objects.equals(out.remove("rowOffset"),"0"))throw denied("当前总体查询不允许分页偏移"); if(out.containsKey("rowOffset")&&!Objects.equals(out.remove("rowOffset"),"0"))throw denied("当前总体查询不允许分页偏移");
if(out.containsKey("rowOrder")&&(!Set.of("department","group").contains(out.get("view"))||!Objects.equals(out.remove("rowOrder"),"scrapDesc")))throw denied("批量汇总排序方式无效");
for(String key:List.of("extraFilters","exclusions"))if(out.containsKey(key))out.put(key,write(canonical(read(out.get(key))))); for(String key:List.of("extraFilters","exclusions"))if(out.containsKey(key))out.put(key,write(canonical(read(out.get(key)))));
if(out.containsKey("columnFilters"))out.put("columnFilters",write(canonical(cn.yunsheng.quality.filters.SourceColumnQueries.parse(out.get("columnFilters"))))); if(out.containsKey("columnFilters"))out.put("columnFilters",write(canonical(cn.yunsheng.quality.filters.SourceColumnQueries.parse(out.get("columnFilters")))));
return out; return out;
@@ -37,7 +37,7 @@ public class AnalysisReportService {
private static final int MAX_DUE_PLANS_PER_TICK = 10; private static final int MAX_DUE_PLANS_PER_TICK = 10;
private static final int MAX_REPORT_SUBMISSIONS_PER_TICK = 10; private static final int MAX_REPORT_SUBMISSIONS_PER_TICK = 10;
private static final long REPORT_LEASE_MINUTES = 5; private static final long REPORT_LEASE_MINUTES = 5;
private static final Set<String> READER_ROLES = Set.of("SYSTEM_ADMIN", "QUALITY_VIEWER", "BUSINESS_ANALYST"); private static final Set<String> READER_ROLES = Set.of("SYSTEM_ADMIN", "DATA_ADMIN", "USER");
private final JdbcTemplate jdbc; private final JdbcTemplate jdbc;
private final AnalysisTurnService turns; private final AnalysisTurnService turns;
private final JdbcAppUserDetailsService users; private final JdbcAppUserDetailsService users;
@@ -17,7 +17,7 @@ import org.springframework.web.bind.annotation.RestController;
/** Periodic reports are exposed alongside the analysis turns they create. */ /** Periodic reports are exposed alongside the analysis turns they create. */
@RestController @RestController
@RequestMapping("/api/analysis") @RequestMapping("/api/analysis")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','BUSINESS_ANALYST','QUALITY_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public class AnalysisReportsController { public class AnalysisReportsController {
private final AnalysisReportService reports; private final AnalysisReportService reports;
@@ -19,9 +19,9 @@ import org.springframework.security.core.userdetails.UsernameNotFoundException;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.web.server.ResponseStatusException; import org.springframework.web.server.ResponseStatusException;
/** Short-lived, in-memory delegation for one native NAO turn. */ /** Short-lived, in-memory delegation for one native DSH turn. */
@Service @Service
public class NaoRuntimeAccessService { public class AnalysisRuntimeAccessService {
private static final Duration TOKEN_TTL = AnalysisRuntimePolicy.DELEGATION_TTL; private static final Duration TOKEN_TTL = AnalysisRuntimePolicy.DELEGATION_TTL;
private static final SecureRandom RANDOM = new SecureRandom(); private static final SecureRandom RANDOM = new SecureRandom();
@@ -36,12 +36,12 @@ public class NaoRuntimeAccessService {
public void setProtocol(AnalysisProtocolService protocol) { this.protocol=protocol; } public void setProtocol(AnalysisProtocolService protocol) { this.protocol=protocol; }
@Autowired @Autowired
public NaoRuntimeAccessService(JdbcTemplate jdbc, JdbcAppUserDetailsService users, public AnalysisRuntimeAccessService(JdbcTemplate jdbc, JdbcAppUserDetailsService users,
GovernanceAccessService governance) { GovernanceAccessService governance) {
this(jdbc, users, governance, Clock.systemUTC()); this(jdbc, users, governance, Clock.systemUTC());
} }
NaoRuntimeAccessService(JdbcTemplate jdbc, JdbcAppUserDetailsService users, AnalysisRuntimeAccessService(JdbcTemplate jdbc, JdbcAppUserDetailsService users,
GovernanceAccessService governance, Clock clock) { GovernanceAccessService governance, Clock clock) {
this.jdbc = jdbc; this.jdbc = jdbc;
this.users = users; this.users = users;
@@ -122,7 +122,7 @@ public class NaoRuntimeAccessService {
private TurnState turnState(String turnId, UUID actorId) { private TurnState turnState(String turnId, UUID actorId) {
return jdbc.query("SELECT status,cancel_requested,auth_scope_hash FROM analysis_turn WHERE id=? AND user_id=?", return jdbc.query("SELECT status,cancel_requested,auth_scope_hash FROM analysis_turn WHERE id=? AND user_id=?",
(rs, row) -> new TurnState(rs.getString(1), rs.getBoolean(2), rs.getString(3)), turnId, actorId.toString()) (rs, row) -> new TurnState(rs.getString(1), rs.getBoolean(2), rs.getString(3)), turnId, actorId.toString())
.stream().findFirst().orElseThrow(NaoRuntimeAccessService::denied); .stream().findFirst().orElseThrow(AnalysisRuntimeAccessService::denied);
} }
public void revokeTurn(String turnId) { public void revokeTurn(String turnId) {
@@ -139,7 +139,7 @@ public class NaoRuntimeAccessService {
private AppUserPrincipal freshPrincipal(UUID actorId) { private AppUserPrincipal freshPrincipal(UUID actorId) {
String username = jdbc.query("SELECT username FROM app_user WHERE id=? AND active=TRUE", String username = jdbc.query("SELECT username FROM app_user WHERE id=? AND active=TRUE",
(rs, row) -> rs.getString(1), actorId.toString()).stream().findFirst() (rs, row) -> rs.getString(1), actorId.toString()).stream().findFirst()
.orElseThrow(NaoRuntimeAccessService::denied); .orElseThrow(AnalysisRuntimeAccessService::denied);
try { try {
UserDetails details = users.loadUserByUsername(username); UserDetails details = users.loadUserByUsername(username);
if (!(details instanceof AppUserPrincipal principal) || !principal.isEnabled()) throw denied(); if (!(details instanceof AppUserPrincipal principal) || !principal.isEnabled()) throw denied();
@@ -162,7 +162,7 @@ public class NaoRuntimeAccessService {
} }
private static ResponseStatusException denied() { private static ResponseStatusException denied() {
return new ResponseStatusException(HttpStatus.FORBIDDEN, "NAO 查询授权已失效"); return new ResponseStatusException(HttpStatus.FORBIDDEN, "DSH 查询授权已失效");
} }
public record Delegation(AppUserPrincipal actor, String releaseId, String turnId, String attemptId) { public record Delegation(AppUserPrincipal actor, String releaseId, String turnId, String attemptId) {
@@ -0,0 +1,56 @@
package cn.yunsheng.quality.analysis;
import java.util.LinkedHashMap;
import java.util.Map;
/** Server-owned analysis runtime identifiers. These values are never accepted from client input. */
final class AnalysisRuntimeEngine {
static final String DSH = "DSH";
static final String HISTORICAL = "HISTORICAL";
static final String INSPECTION = "CONTROLLED_INSPECTION_PROTOCOL";
private AnalysisRuntimeEngine() {}
static boolean canExecute(Map<String, Object> plan) {
String engine = fromPlan(plan);
return DSH.equals(engine) || INSPECTION.equals(engine);
}
static String require(Object value) {
if (value == null || DSH.equalsIgnoreCase(value.toString().strip())) return DSH;
throw new IllegalArgumentException("Unsupported analysis runtime engine");
}
/** Saved plans keep provenance. Earlier engine identifiers are displayed as historical. */
static String fromPlan(Map<String, Object> plan) {
if (plan == null) return HISTORICAL;
Object engine = plan.get("engine");
if (DSH.equals(engine) || INSPECTION.equals(engine) || "CONTROLLED".equals(engine)) return (String) engine;
if (engine == null && ("NATIVE_DSH".equals(plan.get("status")) || "NATIVE_DSH_PENDING".equals(plan.get("status")))) return DSH;
return HISTORICAL;
}
static boolean isRuntimePlan(Map<String, Object> plan) {
if (plan == null) return false;
return DSH.equals(plan.get("engine"))
|| Boolean.TRUE.equals(plan.get("nativeRuntime"))
|| (plan.get("status") instanceof String status && status.startsWith("NATIVE_"));
}
static Map<String, Object> publicMetadata(Map<String, Object> source) {
var result = new LinkedHashMap<>(source);
if (source.containsKey("engine")) result.put("engine", fromPlan(source));
if (HISTORICAL.equals(fromPlan(source)) && source.get("status") instanceof String status && status.startsWith("NATIVE_"))
result.put("status", HISTORICAL);
result.remove("runtimeEngine");
return result;
}
static String pendingStatus(String engine) { return "NATIVE_" + require(engine) + "_PENDING"; }
static String runningStatus(String engine) { return "NATIVE_" + require(engine); }
static String progressLabel(String engine) { return require(engine) + " 正在处理"; }
static String runtimeName(String engine) { return require(engine) + " runtime"; }
}
@@ -26,17 +26,17 @@ import org.springframework.web.server.ResponseStatusException;
import org.springframework.web.util.UriComponentsBuilder; import org.springframework.web.util.UriComponentsBuilder;
import jakarta.servlet.http.HttpServletRequest; import jakarta.servlet.http.HttpServletRequest;
/** Narrow local bridge used only by the delegated native NAO runtime. */ /** Narrow local bridge used only by the delegated native DSH runtime. */
@RestController @RestController
@RequestMapping("/api/analysis/runtime") @RequestMapping("/api/analysis/runtime")
public class NaoRuntimeQueryController { public class AnalysisRuntimeQueryController {
private static final Set<String> METRIC_QUERY = Set.of("year", "month", "monthStart", "monthEnd", "quarter", "department", "group", private static final Set<String> METRIC_QUERY = Set.of("year", "month", "monthStart", "monthEnd", "quarter", "department", "group",
"product", "projectGroup", "extraFilters", "columnFilters", "exclusions", "view", "includeUnassigned", "releaseId", "trendGranularity", "product", "projectGroup", "extraFilters", "columnFilters", "exclusions", "view", "includeUnassigned", "releaseId", "trendGranularity",
"rowLimit", "rowOffset", "rowOrder"); "rowLimit", "rowOffset", "rowOrder");
private static final Set<String> TOP_QUERY = Set.of("stage", "dimension", "category", "releaseId", "year", private static final Set<String> TOP_QUERY = Set.of("stage", "dimension", "category", "releaseId", "year",
"month", "quarter", "department", "group", "product", "projectGroup", "extraFilters", "includeUnassigned", "limit"); "month", "quarter", "department", "group", "product", "projectGroup", "extraFilters", "includeUnassigned", "limit");
private final NaoRuntimeAccessService access; private final AnalysisRuntimeAccessService access;
private final ReleaseController releases; private final ReleaseController releases;
private final OptionsController options; private final OptionsController options;
private final MetricsController metrics; private final MetricsController metrics;
@@ -56,7 +56,7 @@ public class NaoRuntimeQueryController {
@org.springframework.beans.factory.annotation.Autowired(required=false) @org.springframework.beans.factory.annotation.Autowired(required=false)
public void setChanges(AnalysisChangesController changes){this.changes=changes;} public void setChanges(AnalysisChangesController changes){this.changes=changes;}
public NaoRuntimeQueryController(NaoRuntimeAccessService access, ReleaseController releases, public AnalysisRuntimeQueryController(AnalysisRuntimeAccessService access, ReleaseController releases,
OptionsController options, MetricsController metrics, OptionsController options, MetricsController metrics,
AnalysisToolsController tools, AnalyticsController analytics, AnalysisToolsController tools, AnalyticsController analytics,
AnalysisFineDefectsController defects, AnalysisInvestigationController investigation) { AnalysisFineDefectsController defects, AnalysisInvestigationController investigation) {
@@ -73,8 +73,8 @@ public class NaoRuntimeQueryController {
@PostMapping("/query") @PostMapping("/query")
public Object query(@RequestBody RuntimeQuery request, HttpServletRequest servletRequest) { public Object query(@RequestBody RuntimeQuery request, HttpServletRequest servletRequest) {
requireLoopback(servletRequest); requireLoopback(servletRequest);
if (request == null) throw bad("NAO 查询请求无效"); if (request == null) throw bad("DSH 查询请求无效");
NaoRuntimeAccessService.Delegation delegation = access.authorize(request.token()); AnalysisRuntimeAccessService.Delegation delegation = access.authorize(request.token());
ParsedTarget target = parseTarget(request.method(), request.path()); ParsedTarget target = parseTarget(request.method(), request.path());
if ("GET".equals(target.method()) && request.payload() != null) throw bad("GET 查询不能包含请求体"); if ("GET".equals(target.method()) && request.payload() != null) throw bad("GET 查询不能包含请求体");
if ("POST".equals(target.method()) && !target.query().isEmpty()) throw bad("POST 查询不能包含 URL 参数"); if ("POST".equals(target.method()) && !target.query().isEmpty()) throw bad("POST 查询不能包含 URL 参数");
@@ -96,7 +96,7 @@ public class NaoRuntimeQueryController {
} }
private Object dispatch(ParsedTarget target, Map<String, Object> payload, private Object dispatch(ParsedTarget target, Map<String, Object> payload,
NaoRuntimeAccessService.Delegation delegation) { AnalysisRuntimeAccessService.Delegation delegation) {
String releaseId = delegation.releaseId(); String releaseId = delegation.releaseId();
return switch (target.method() + " " + target.path()) { return switch (target.method() + " " + target.path()) {
case "POST /api/analysis/protocol/history" -> { case "POST /api/analysis/protocol/history" -> {
@@ -127,7 +127,7 @@ public class NaoRuntimeQueryController {
case "GET /api/options" -> { case "GET /api/options" -> {
rejectUnknown(target.query(), Set.of("releaseId")); rejectUnknown(target.query(), Set.of("releaseId"));
requireBoundRelease(target.query().get("releaseId"), releaseId); requireBoundRelease(target.query().get("releaseId"), releaseId);
yield options.options(releaseId, delegation.actor()); yield AnalysisAgentCatalog.active(options.options(releaseId, delegation.actor()));
} }
case "GET /api/metrics" -> metricQuery(target.query(), releaseId, delegation); case "GET /api/metrics" -> metricQuery(target.query(), releaseId, delegation);
case "GET /api/analysis/tools/catalog" -> { case "GET /api/analysis/tools/catalog" -> {
@@ -163,7 +163,7 @@ public class NaoRuntimeQueryController {
Map<String, Object> bound = boundPayload(payload, releaseId); Map<String, Object> bound = boundPayload(payload, releaseId);
yield defects.query(bound, delegation.actor()); yield defects.query(bound, delegation.actor());
} }
default -> throw new ResponseStatusException(HttpStatus.NOT_FOUND, "NAO 查询路由未开放"); default -> throw new ResponseStatusException(HttpStatus.NOT_FOUND, "DSH 查询路由未开放");
}; };
} }
@@ -178,7 +178,7 @@ public class NaoRuntimeQueryController {
} }
private Object metricQuery(Map<String, String> query, String releaseId, private Object metricQuery(Map<String, String> query, String releaseId,
NaoRuntimeAccessService.Delegation delegation) { AnalysisRuntimeAccessService.Delegation delegation) {
rejectUnknown(query, METRIC_QUERY); rejectUnknown(query, METRIC_QUERY);
requireBoundRelease(query.get("releaseId"), releaseId); requireBoundRelease(query.get("releaseId"), releaseId);
if(query.containsKey("monthStart")||query.containsKey("monthEnd"))return metrics.calculateRange( if(query.containsKey("monthStart")||query.containsKey("monthEnd"))return metrics.calculateRange(
@@ -199,7 +199,7 @@ public class NaoRuntimeQueryController {
} }
private Object topQuery(Map<String, String> query, String releaseId, private Object topQuery(Map<String, String> query, String releaseId,
NaoRuntimeAccessService.Delegation delegation) { AnalysisRuntimeAccessService.Delegation delegation) {
rejectUnknown(query, TOP_QUERY); rejectUnknown(query, TOP_QUERY);
requireBoundRelease(query.get("releaseId"), releaseId); requireBoundRelease(query.get("releaseId"), releaseId);
return analytics.top(required(query, "stage"), query.getOrDefault("dimension", "category"), return analytics.top(required(query, "stage"), query.getOrDefault("dimension", "category"),
@@ -210,7 +210,7 @@ public class NaoRuntimeQueryController {
} }
private Map<String, Object> boundPayload(Map<String, Object> payload, String releaseId) { private Map<String, Object> boundPayload(Map<String, Object> payload, String releaseId) {
if (payload == null) throw bad("NAO 查询参数无效"); if (payload == null) throw bad("DSH 查询参数无效");
Object requested = payload.get("releaseId"); Object requested = payload.get("releaseId");
requireBoundRelease(requested == null ? null : requested.toString(), releaseId); requireBoundRelease(requested == null ? null : requested.toString(), releaseId);
Map<String, Object> bound = new LinkedHashMap<>(payload); Map<String, Object> bound = new LinkedHashMap<>(payload);
@@ -220,31 +220,31 @@ public class NaoRuntimeQueryController {
private ParsedTarget parseTarget(String method, String rawPath) { private ParsedTarget parseTarget(String method, String rawPath) {
if (method == null || rawPath == null || rawPath.length() > 4096 || !rawPath.startsWith("/") if (method == null || rawPath == null || rawPath.length() > 4096 || !rawPath.startsWith("/")
|| rawPath.startsWith("//") || rawPath.contains("#")) throw bad("NAO 查询目标无效"); || rawPath.startsWith("//") || rawPath.contains("#")) throw bad("DSH 查询目标无效");
String normalizedMethod = method.toUpperCase(java.util.Locale.ROOT); String normalizedMethod = method.toUpperCase(java.util.Locale.ROOT);
if (!List.of("GET", "POST").contains(normalizedMethod)) throw bad("NAO 查询方法未开放"); if (!List.of("GET", "POST").contains(normalizedMethod)) throw bad("DSH 查询方法未开放");
URI uri; URI uri;
try { uri = URI.create(rawPath); } try { uri = URI.create(rawPath); }
catch (RuntimeException invalid) { throw bad("NAO 查询目标无效"); } catch (RuntimeException invalid) { throw bad("DSH 查询目标无效"); }
if (uri.isAbsolute() || uri.getRawUserInfo() != null || uri.getRawAuthority() != null) if (uri.isAbsolute() || uri.getRawUserInfo() != null || uri.getRawAuthority() != null)
throw bad("NAO 查询目标无效"); throw bad("DSH 查询目标无效");
String path = uri.getRawPath(); String path = uri.getRawPath();
if (path == null || !path.startsWith("/") || path.contains("%") || path.contains("..") || path.contains("\\")) if (path == null || !path.startsWith("/") || path.contains("%") || path.contains("..") || path.contains("\\"))
throw bad("NAO 查询路径无效"); throw bad("DSH 查询路径无效");
Map<String, String> query = new LinkedHashMap<>(); Map<String, String> query = new LinkedHashMap<>();
String rawQuery = uri.getRawQuery(); String rawQuery = uri.getRawQuery();
if (rawQuery != null && !rawQuery.isEmpty()) for (String pair : rawQuery.split("&", -1)) { if (rawQuery != null && !rawQuery.isEmpty()) for (String pair : rawQuery.split("&", -1)) {
int separator = pair.indexOf('='); int separator = pair.indexOf('=');
String key = decode(separator < 0 ? pair : pair.substring(0, separator)); String key = decode(separator < 0 ? pair : pair.substring(0, separator));
String value = decode(separator < 0 ? "" : pair.substring(separator + 1)); String value = decode(separator < 0 ? "" : pair.substring(separator + 1));
if (key.isBlank() || query.putIfAbsent(key, value) != null) throw bad("NAO 查询参数重复或无效"); if (key.isBlank() || query.putIfAbsent(key, value) != null) throw bad("DSH 查询参数重复或无效");
} }
return new ParsedTarget(normalizedMethod, path, Map.copyOf(query)); return new ParsedTarget(normalizedMethod, path, Map.copyOf(query));
} }
private String decode(String raw) { private String decode(String raw) {
try { return URLDecoder.decode(raw, StandardCharsets.UTF_8); } try { return URLDecoder.decode(raw, StandardCharsets.UTF_8); }
catch (IllegalArgumentException invalid) { throw bad("NAO 查询参数编码无效"); } catch (IllegalArgumentException invalid) { throw bad("DSH 查询参数编码无效"); }
} }
private <T> T asActor(cn.yunsheng.quality.identity.AppUserPrincipal actor, private <T> T asActor(cn.yunsheng.quality.identity.AppUserPrincipal actor,
@@ -261,33 +261,33 @@ public class NaoRuntimeQueryController {
private void requireLoopback(HttpServletRequest request) { private void requireLoopback(HttpServletRequest request) {
try { try {
InetAddress remote = InetAddress.getByName(request.getRemoteAddr()); InetAddress remote = InetAddress.getByName(request.getRemoteAddr());
if (!remote.isLoopbackAddress()) throw new ResponseStatusException(HttpStatus.FORBIDDEN, "NAO 查询仅允许本机访问"); if (!remote.isLoopbackAddress()) throw new ResponseStatusException(HttpStatus.FORBIDDEN, "DSH 查询仅允许本机访问");
} catch (java.net.UnknownHostException invalid) { } catch (java.net.UnknownHostException invalid) {
throw new ResponseStatusException(HttpStatus.FORBIDDEN, "NAO 查询仅允许本机访问"); throw new ResponseStatusException(HttpStatus.FORBIDDEN, "DSH 查询仅允许本机访问");
} }
} }
private void requireBoundRelease(String requested, String bound) { private void requireBoundRelease(String requested, String bound) {
if (requested != null && !requested.isBlank() && !bound.equals(requested)) if (requested != null && !requested.isBlank() && !bound.equals(requested))
throw new ResponseStatusException(HttpStatus.FORBIDDEN, "NAO 查询不能切换固定发布版本"); throw new ResponseStatusException(HttpStatus.FORBIDDEN, "DSH 查询不能切换固定发布版本");
} }
private void rejectUnknown(Map<String, ?> values, Set<String> allowed) { private void rejectUnknown(Map<String, ?> values, Set<String> allowed) {
if (!allowed.containsAll(values.keySet())) throw bad("NAO 查询包含未开放参数"); if (!allowed.containsAll(values.keySet())) throw bad("DSH 查询包含未开放参数");
} }
private String required(Map<String, String> values, String key) { private String required(Map<String, String> values, String key) {
String value = values.get(key); String value = values.get(key);
if (value == null || value.isBlank()) throw bad("NAO 查询缺少必要参数"); if (value == null || value.isBlank()) throw bad("DSH 查询缺少必要参数");
return value; return value;
} }
private Integer integer(Map<String, String> values, String key) { private Integer integer(Map<String, String> values, String key) {
String raw = values.get(key); String raw = values.get(key);
if (raw == null || raw.isBlank()) return null; if (raw == null || raw.isBlank()) return null;
if (!raw.matches("-?\\d{1,5}")) throw bad("NAO 查询数字参数无效"); if (!raw.matches("-?\\d{1,5}")) throw bad("DSH 查询数字参数无效");
try { return Integer.valueOf(raw); } try { return Integer.valueOf(raw); }
catch (NumberFormatException invalid) { throw bad("NAO 查询数字参数无效"); } catch (NumberFormatException invalid) { throw bad("DSH 查询数字参数无效"); }
} }
private boolean bool(Map<String, String> values, String key, boolean fallback) { private boolean bool(Map<String, String> values, String key, boolean fallback) {
@@ -295,7 +295,7 @@ public class NaoRuntimeQueryController {
if (raw == null || raw.isBlank()) return fallback; if (raw == null || raw.isBlank()) return fallback;
if ("true".equalsIgnoreCase(raw)) return true; if ("true".equalsIgnoreCase(raw)) return true;
if ("false".equalsIgnoreCase(raw)) return false; if ("false".equalsIgnoreCase(raw)) return false;
throw bad("NAO 查询布尔参数无效"); throw bad("DSH 查询布尔参数无效");
} }
private ResponseStatusException bad(String message) { private ResponseStatusException bad(String message) {
@@ -11,7 +11,7 @@ import org.springframework.web.bind.annotation.*;
/** Authorized source schema and projected member lists; never accepts SQL. */ /** Authorized source schema and projected member lists; never accepts SQL. */
@RestController @RestController
@RequestMapping("/api/analysis/source") @RequestMapping("/api/analysis/source")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','BUSINESS_ANALYST','QUALITY_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public class AnalysisSourceQueryController { public class AnalysisSourceQueryController {
private final SourceColumnQueries columns; private final SourceColumnQueries columns;
private final ReleaseService releases; private final ReleaseService releases;
@@ -43,12 +43,16 @@ public class AnalysisSourceQueryController {
@PostMapping("/values") @PostMapping("/values")
public Map<String,Object> values(@RequestBody Map<String,Object> input,@AuthenticationPrincipal AppUserPrincipal actor) { public Map<String,Object> values(@RequestBody Map<String,Object> input,@AuthenticationPrincipal AppUserPrincipal actor) {
String scope=captureScope(actor); String scope=captureScope(actor);
if(input==null||!Set.of("releaseId","period","filters","field","stages","search","limit","offset").containsAll(input.keySet()))throw bad("候选值请求包含未知字段"); if(input==null||!Set.of("releaseId","period","filters","field","stages","search","limit","offset","candidateOnly").containsAll(input.keySet()))throw bad("候选值请求包含未知字段");
String release=releases.resolvePublishedRelease(Objects.toString(input.get("releaseId"),null)); String release=releases.resolvePublishedRelease(Objects.toString(input.get("releaseId"),null));
Map<String,Object> context=context(input); Map<String,Object> context=context(input);
if(!(input.get("field") instanceof String field)||field.isBlank()||input.get("search")!=null&&!(input.get("search") instanceof String))throw bad("候选字段或搜索值无效"); if(!(input.get("field") instanceof String field)||field.isBlank()||input.get("search")!=null&&!(input.get("search") instanceof String))throw bad("候选字段或搜索值无效");
Map<String,Object> result=new LinkedHashMap<>(columns.values(release,context,field,strings(input.get("stages"),2),Objects.toString(input.get("search"),null), if(input.containsKey("candidateOnly")&&!(input.get("candidateOnly") instanceof Boolean))throw bad("候选分页参数无效");
integer(input.getOrDefault("limit",50),1,200),integer(input.getOrDefault("offset",0),0,100000),actor)); List<String> stages=strings(input.get("stages"),2);String search=Objects.toString(input.get("search"),null);
int limit=integer(input.getOrDefault("limit",50),1,200),offset=integer(input.getOrDefault("offset",0),0,100000);
Map<String,Object> result=new LinkedHashMap<>(Boolean.TRUE.equals(input.get("candidateOnly"))
? columns.values(release,context,field,stages,search,limit,offset,true,actor)
: columns.values(release,context,field,stages,search,limit,offset,actor));
result.put("period",input.get("period"));result.put("filters",input.get("filters")); result.put("period",input.get("period"));result.put("filters",input.get("filters"));
result.put("values",AnalysisProtocolService.list(result.get("rows")).stream().map(row->AnalysisProtocolService.map(row).get("value")).toList()); result.put("values",AnalysisProtocolService.list(result.get("rows")).stream().map(row->AnalysisProtocolService.map(row).get("value")).toList());
result.put("candidateOnly",true);reauthorize(actor,scope,false);return result; result.put("candidateOnly",true);reauthorize(actor,scope,false);return result;
@@ -11,7 +11,7 @@ import org.springframework.web.server.ResponseStatusException;
@RestController @RestController
@RequestMapping("/api/analysis/tools") @RequestMapping("/api/analysis/tools")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','BUSINESS_ANALYST','QUALITY_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public class AnalysisToolsController { public class AnalysisToolsController {
private final AnalysisExplorationTools tools; private final AnalysisExplorationTools tools;
public AnalysisToolsController(AnalysisExplorationTools tools) { this.tools = tools; } public AnalysisToolsController(AnalysisExplorationTools tools) { this.tools = tools; }
@@ -58,8 +58,8 @@ public class AnalysisTurnService {
private static final int TOOL_CALLS_MAX = 12; private static final int TOOL_CALLS_MAX = 12;
private static final int PLAN_REVISIONS_MAX = 8; private static final int PLAN_REVISIONS_MAX = 8;
private static final int DEADLINE_SECONDS = 240; private static final int DEADLINE_SECONDS = 240;
private static final int NAO_DEADLINE_SECONDS = Math.toIntExact(AnalysisRuntimePolicy.WATCHDOG_TIMEOUT.toSeconds()); private static final int RUNTIME_DEADLINE_SECONDS = Math.toIntExact(AnalysisRuntimePolicy.WATCHDOG_TIMEOUT.toSeconds());
private static final int NAO_TOOL_CALLS_MAX = 1000; private static final int RUNTIME_TOOL_CALLS_MAX = 1000;
private static final int NATIVE_HISTORY_QUERY_LIMIT = 12; private static final int NATIVE_HISTORY_QUERY_LIMIT = 12;
private static final int NATIVE_HISTORY_ENTRY_LIMIT = 8; private static final int NATIVE_HISTORY_ENTRY_LIMIT = 8;
private static final int NATIVE_HISTORY_CONTEXT_LIMIT = 16_000; private static final int NATIVE_HISTORY_CONTEXT_LIMIT = 16_000;
@@ -75,11 +75,13 @@ public class AnalysisTurnService {
private final GovernanceAccessService governance; private final GovernanceAccessService governance;
private final MetricsResponseAccess responseAccess; private final MetricsResponseAccess responseAccess;
private final AnalysisExplorationTools exploration; private final AnalysisExplorationTools exploration;
private final NaoAnalysisRunner naoRunner; private final DshAnalysisRunner dshRunner;
private final NaoRuntimeAccessService naoAccess; private final AnalysisRuntimeAccessService runtimeAccess;
private AnalysisProtocolService protocol; private AnalysisProtocolService protocol;
@org.springframework.beans.factory.annotation.Autowired @org.springframework.beans.factory.annotation.Autowired
public void setProtocol(AnalysisProtocolService protocol){this.protocol=protocol;} public void setProtocol(AnalysisProtocolService protocol){this.protocol=protocol;}
private final Map<String, String> runtimeDrafts = new java.util.concurrent.ConcurrentHashMap<>();
private final Map<String, Map<String, Object>> runtimeModelOutputs = new java.util.concurrent.ConcurrentHashMap<>();
@org.springframework.beans.factory.annotation.Value("${quality.analysis.visuals-enabled:true}") @org.springframework.beans.factory.annotation.Value("${quality.analysis.visuals-enabled:true}")
private boolean visualsEnabled = true; private boolean visualsEnabled = true;
private final ThreadPoolExecutor executor = new ThreadPoolExecutor(2, 2, 0, TimeUnit.MILLISECONDS, private final ThreadPoolExecutor executor = new ThreadPoolExecutor(2, 2, 0, TimeUnit.MILLISECONDS,
@@ -100,13 +102,13 @@ public class AnalysisTurnService {
@org.springframework.beans.factory.annotation.Autowired @org.springframework.beans.factory.annotation.Autowired
public AnalysisTurnService(JdbcTemplate jdbc, JsonMapper json, MetricsService metrics, AnalyticsController analytics, AnalysisModelGateway model, public AnalysisTurnService(JdbcTemplate jdbc, JsonMapper json, MetricsService metrics, AnalyticsController analytics, AnalysisModelGateway model,
JdbcAppUserDetailsService users, GovernanceAccessService governance, MetricsResponseAccess responseAccess, JdbcAppUserDetailsService users, GovernanceAccessService governance, MetricsResponseAccess responseAccess,
AnalysisExplorationTools exploration, NaoAnalysisRunner naoRunner, AnalysisExplorationTools exploration, DshAnalysisRunner dshRunner,
NaoRuntimeAccessService naoAccess) { AnalysisRuntimeAccessService runtimeAccess) {
this.jdbc = jdbc; this.json = json; this.metrics = metrics; this.analytics = analytics; this.model = model; this.users = users; this.governance = governance; this.jdbc = jdbc; this.json = json; this.metrics = metrics; this.analytics = analytics; this.model = model; this.users = users; this.governance = governance;
this.responseAccess = responseAccess; this.responseAccess = responseAccess;
this.exploration = exploration; this.exploration = exploration;
this.naoRunner = naoRunner; this.dshRunner = dshRunner;
this.naoAccess = naoAccess; this.runtimeAccess = runtimeAccess;
} }
@EventListener(ApplicationReadyEvent.class) @EventListener(ApplicationReadyEvent.class)
@@ -131,7 +133,8 @@ public class AnalysisTurnService {
@Transactional @Transactional
public Map<String, Object> createSession(CreateSessionRequest request, AppUserPrincipal actor) { public Map<String, Object> createSession(CreateSessionRequest request, AppUserPrincipal actor) {
requireAnalysisAccess(actor); requireAnalysisAccess(actor);
if(protocol!=null&&protocol.enabled()&&naoRunner==null)throw conflict("分析服务尚未就绪,请稍后新建会话"); if (protocol != null && (!protocol.enabled() || dshRunner == null))
throw conflict("DSH 分析协议尚未启用,请稍后新建会话");
Map<String, Object> context = sanitizeContext(request == null ? null : request.context()); Map<String, Object> context = sanitizeContext(request == null ? null : request.context());
String releaseId = request == null ? null : clean(request.releaseId()); String releaseId = request == null ? null : clean(request.releaseId());
if (releaseId == null) releaseId = clean(context.get("releaseId")); if (releaseId == null) releaseId = clean(context.get("releaseId"));
@@ -145,7 +148,7 @@ public class AnalysisTurnService {
jdbc.update("INSERT INTO analysis_session(id,user_id,title,release_id,auth_scope_hash,context_json) VALUES(?,?,?,?,?,CAST(? AS JSON))", jdbc.update("INSERT INTO analysis_session(id,user_id,title,release_id,auth_scope_hash,context_json) VALUES(?,?,?,?,?,CAST(? AS JSON))",
id, actor.getId().toString(), title == null ? "新质量分析" : cap(title, 160), releaseId, id, actor.getId().toString(), title == null ? "新质量分析" : cap(title, 160), releaseId,
scopeHash(actor), write(context)); scopeHash(actor), write(context));
if(protocol!=null&&protocol.enabled()&&naoRunner!=null)protocol.initialize(id,context); if(protocol!=null&&protocol.enabled()&&dshRunner!=null)protocol.initialize(id,context);
return Map.of("session", Map.of("id", id, "releaseId", releaseId, "context", context, "status", "ACTIVE", return Map.of("session", Map.of("id", id, "releaseId", releaseId, "context", context, "status", "ACTIVE",
"createdAt", Instant.now().toString())); "createdAt", Instant.now().toString()));
} }
@@ -230,7 +233,7 @@ public class AnalysisTurnService {
@Transactional @Transactional
public Map<String, Object> createTurn(String sessionId, TurnRequest request, AppUserPrincipal actor) { public Map<String, Object> createTurn(String sessionId, TurnRequest request, AppUserPrincipal actor) {
requireAnalysisAccess(actor); requireAnalysisAccess(actor);
if (naoRunner != null) governance.requireBoundAction("MODEL", "model.intent.parse"); if (dshRunner != null) governance.requireBoundAction("MODEL", "model.intent.parse");
if (request == null || request.question() == null || request.question().isBlank() || request.question().length() > 1600) if (request == null || request.question() == null || request.question().isBlank() || request.question().length() > 1600)
throw bad("请输入不超过1600字的质量分析问题"); throw bad("请输入不超过1600字的质量分析问题");
String question = request.question().strip(); String question = request.question().strip();
@@ -242,6 +245,7 @@ public class AnalysisTurnService {
if (requestedReleaseId != null && !session.releaseId().equals(requestedReleaseId)) if (requestedReleaseId != null && !session.releaseId().equals(requestedReleaseId))
throw conflict("会话已固定数据版本,请新建会话后切换版本"); throw conflict("会话已固定数据版本,请新建会话后切换版本");
boolean unified=protocol!=null&&protocol.ownsSession(sessionId); boolean unified=protocol!=null&&protocol.ownsSession(sessionId);
if (protocol != null && !unified) throw conflict("DSH 会话缺少有效的协议绑定,请新建会话");
Map<String, Object> context = new LinkedHashMap<>(unified?protocol.baseContext(sessionId):session.context()); Map<String, Object> context = new LinkedHashMap<>(unified?protocol.baseContext(sessionId):session.context());
if(!unified){merge(context, submittedContext);bindResultReference(context, request.context(), session.releaseId(), actor);} if(!unified){merge(context, submittedContext);bindResultReference(context, request.context(), session.releaseId(), actor);}
String requestedRelease = clean(context.get("releaseId")); String requestedRelease = clean(context.get("releaseId"));
@@ -260,16 +264,18 @@ public class AnalysisTurnService {
} }
Integer active = jdbc.queryForObject("SELECT COUNT(*) FROM analysis_turn WHERE session_id=? AND status IN ('QUEUED','RUNNING')", Integer.class, sessionId); Integer active = jdbc.queryForObject("SELECT COUNT(*) FROM analysis_turn WHERE session_id=? AND status IN ('QUEUED','RUNNING')", Integer.class, sessionId);
if (active != null && active > 0) throw conflict("当前会话仍有分析任务,请等待完成或停止后继续"); if (active != null && active > 0) throw conflict("当前会话仍有分析任务,请等待完成或停止后继续");
Map<String, Object> task = naoRunner == null Map<String, Object> task = dshRunner == null
? AnalysisTaskState.resolve(question, mapValue(context.get("_task")), submittedContext) ? AnalysisTaskState.resolve(question, mapValue(context.get("_task")), submittedContext)
: new LinkedHashMap<>(); : new LinkedHashMap<>();
context.put("_task", task); context.put("_task", task);
String runtimeEngine = AnalysisRuntimeEngine.DSH;
String turnId = UUID.randomUUID().toString(); String turnId = UUID.randomUUID().toString();
Map<String, Object> budget = budget(); Map<String, Object> budget = budget();
Map<String, Object> plan = new LinkedHashMap<>(); Map<String, Object> plan = new LinkedHashMap<>();
plan.put("status", "PENDING_INTENT"); plan.put("steps", List.of()); plan.put("maxPlanRevisions", PLAN_REVISIONS_MAX); plan.put("status", "PENDING_INTENT"); plan.put("steps", List.of()); plan.put("maxPlanRevisions", PLAN_REVISIONS_MAX);
if (naoRunner != null) { if (dshRunner != null) {
plan.put("status", "NATIVE_NAO_PENDING"); plan.put("engine", "NAO"); plan.put("nativeRuntime", true); plan.put("status", AnalysisRuntimeEngine.pendingStatus(runtimeEngine));
plan.put("engine", runtimeEngine); plan.put("nativeRuntime", true);
} }
// Freeze the complete effective context at submission time. Session context is mutable // Freeze the complete effective context at submission time. Session context is mutable
// conversational state and must never silently change the meaning of a retried turn. // conversational state and must never silently change the meaning of a retried turn.
@@ -289,8 +295,11 @@ public class AnalysisTurnService {
} }
throw conflict("分析任务已重复提交"); throw conflict("分析任务已重复提交");
} }
if(unified)protocol.prepare(sessionId,turnId,actor,false); if(unified) {
else appendEvent(turnId, "progress", Map.of("turnId", turnId, "status", "QUEUED", "message", "分析任务已排队")); protocol.prepare(sessionId,turnId,actor,false);
}
else appendEvent(turnId, "progress", Map.of("turnId", turnId, "status", "QUEUED", "message", "分析任务已排队",
"engine", dshRunner == null ? "CONTROLLED" : runtimeEngine));
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() { TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
@Override public void afterCommit() { @Override public void afterCommit() {
try { submitWorker(turnId); } try { submitWorker(turnId); }
@@ -325,12 +334,12 @@ public class AnalysisTurnService {
public Map<String,Object> recoverVisuals(String turnId,AppUserPrincipal submittedActor) { public Map<String,Object> recoverVisuals(String turnId,AppUserPrincipal submittedActor) {
AppUserPrincipal actor=currentUser(submittedActor.getId());requireAnalysisAccess(actor);requireTurn(turnId,actor); AppUserPrincipal actor=currentUser(submittedActor.getId());requireAnalysisAccess(actor);requireTurn(turnId,actor);
if(!visualsEnabled||protocol==null||!protocol.ownsTurn(turnId)||naoRunner==null)throw conflict("该分析暂不支持独立恢复图表"); if(!visualsEnabled||protocol==null||!protocol.ownsTurn(turnId)||dshRunner==null)throw conflict("该分析暂不支持独立恢复图表");
var claim=asSecurityActor(actor,()->protocol.beginVisualRecovery(turnId,actor)); var claim=asSecurityActor(actor,()->protocol.beginVisualRecovery(turnId,actor));
if(Boolean.TRUE.equals(claim.get("claimed"))) { if(Boolean.TRUE.equals(claim.get("claimed"))) {
String id=claim.get("id").toString(),token=claim.get("token").toString(); String id=claim.get("id").toString(),token=claim.get("token").toString();
try { try {
var rebuilt=naoRunner.rebuildVisuals(id,mapValue(claim.get("input"))); var rebuilt=dshRunner.rebuildVisuals(id,mapValue(claim.get("input")));
AppUserPrincipal fresh=currentUser(submittedActor.getId());requireAnalysisAccess(fresh);requireTurn(turnId,fresh); AppUserPrincipal fresh=currentUser(submittedActor.getId());requireAnalysisAccess(fresh);requireTurn(turnId,fresh);
asSecurityActor(fresh,()->{protocol.completeVisualRecovery(id,token,rebuilt,fresh);return null;}); asSecurityActor(fresh,()->{protocol.completeVisualRecovery(id,token,rebuilt,fresh);return null;});
if(!"ready".equals(rebuilt.get("visualState")))throw new ResponseStatusException(HttpStatus.UNPROCESSABLE_ENTITY,"原结果尚不能完整恢复图表,已保留原分析"); if(!"ready".equals(rebuilt.get("visualState")))throw new ResponseStatusException(HttpStatus.UNPROCESSABLE_ENTITY,"原结果尚不能完整恢复图表,已保留原分析");
@@ -363,7 +372,20 @@ public class AnalysisTurnService {
Map<String, Object> result = new LinkedHashMap<>(); Map<String, Object> result = new LinkedHashMap<>();
result.put("id", turn.id()); result.put("turnId", turn.id()); result.put("sessionId", turn.sessionId()); result.put("status", turn.status()); result.put("id", turn.id()); result.put("turnId", turn.id()); result.put("sessionId", turn.sessionId()); result.put("status", turn.status());
result.put("readOnly",protocol!=null&&protocol.legacyReadOnly(turn.sessionId())); result.put("readOnly",protocol!=null&&protocol.legacyReadOnly(turn.sessionId()));
result.put("question", turn.question()); result.put("plan", readNullable(turn.planJson(), Map.of())); result.put("question", turn.question()); result.put("plan", AnalysisRuntimeEngine.publicMetadata(plan));
if (!AnalysisRuntimeEngine.canExecute(plan)) result.put("retryAvailable", false);
Map<String,Object> nativeTask=mapValue(plan.get("task"));
boolean dshNativeTurn=AnalysisRuntimeEngine.DSH.equals(AnalysisRuntimeEngine.fromPlan(plan))
&&("native_agent".equals(answer.get("executionMode"))
||"native_agent".equals(plan.get("executionMode"))
||"native_agent".equals(nativeTask.get("executionMode")));
Object storedTimeline=answer.get("timeline") instanceof List<?> ? answer.get("timeline") : plan.get("timeline");
if(dshNativeTurn&&terminal(turn.status())&&storedTimeline instanceof List<?> timeline)
// Session history restores answer.timeline. Prefer that persisted timeline
// when it contains the final assistant entry; older turns may only have the
// progress timeline in plan.timeline. Keep the same source at turn detail
// and snapshot boundaries so export identity checks compare one snapshot.
result.put("timeline",timeline);
result.put("steps", readAny(turn.stepsJson(), List.of())); result.put("steps", readAny(turn.stepsJson(), List.of()));
try { governance.requireEvidence(actor); result.put("evidence", readAny(turn.evidenceJson(), List.of())); } try { governance.requireEvidence(actor); result.put("evidence", readAny(turn.evidenceJson(), List.of())); }
catch (ResponseStatusException denied) { result.put("evidence", List.of()); result.put("evidenceAvailable", false); } catch (ResponseStatusException denied) { result.put("evidence", List.of()); result.put("evidenceAvailable", false); }
@@ -371,7 +393,7 @@ public class AnalysisTurnService {
result.putAll(visibleVisuals(answer, turn, actor, context,authorizedHistoryReport)); result.putAll(visibleVisuals(answer, turn, actor, context,authorizedHistoryReport));
for (String field : List.of("engine", "explanationEngine", "explanationWarning", "warning", for (String field : List.of("engine", "explanationEngine", "explanationWarning", "warning",
"businessResultsSentToModel", "releaseId", "task", "analysisChecks", "agentDecisions", "completion", "businessResultsSentToModel", "releaseId", "task", "analysisChecks", "agentDecisions", "completion",
"reportMarkdown", "reportHtml", "reportTitle", "nativeChatId", "toolCallCount", "protocol", "queryEvidenceRefs", "actualModel", "phaseTimingsMs", "executionMode", "completionMethod", "readingSummary", "answerQuality", "scopeCoverage", "explorationWarnings", "conclusionProtocolQueryIds", "attemptProtocolQueryIds", "memberList", "recordList", "historySources", "historySections", "historyCoverage", "historyEvidence", "historyVisualSources", "inspectionCoverage", "controlledInspection")) "modelAnswer", "answerOrigin", "answerValidation", "reportMarkdown", "reportHtml", "reportTitle", "nativeChatId", "toolCallCount", "protocol", "queryEvidenceRefs", "actualModel", "phaseTimingsMs", "executionMode", "completionMethod", "readingSummary", "answerQuality", "scopeCoverage", "explorationWarnings", "conclusionProtocolQueryIds", "attemptProtocolQueryIds", "memberList", "recordList", "historySources", "historySections", "historyCoverage", "historyEvidence", "historyVisualSources", "inspectionCoverage", "controlledInspection"))
if (answer.containsKey(field)) result.put(field, answer.get(field)); if (answer.containsKey(field)) result.put(field, answer.get(field));
result.put("stopReason", turn.stopReason()); result.put("budget", budget); result.put("stopReason", turn.stopReason()); result.put("budget", budget);
result.putAll(jdbc.queryForObject("SELECT UNIX_TIMESTAMP(created_at)*1000,UNIX_TIMESTAMP(started_at)*1000,UNIX_TIMESTAMP(finished_at)*1000 FROM analysis_turn WHERE id=? AND user_id=?", result.putAll(jdbc.queryForObject("SELECT UNIX_TIMESTAMP(created_at)*1000,UNIX_TIMESTAMP(started_at)*1000,UNIX_TIMESTAMP(finished_at)*1000 FROM analysis_turn WHERE id=? AND user_id=?",
@@ -379,7 +401,7 @@ public class AnalysisTurnService {
result.put("feedback", feedback(turn.id())); result.put("feedback", feedback(turn.id()));
if(visualsEnabled&&protocol!=null&&protocol.ownsTurn(turn.id())&&Set.of("COMPLETED","PARTIAL").contains(turn.status())) if(visualsEnabled&&protocol!=null&&protocol.ownsTurn(turn.id())&&Set.of("COMPLETED","PARTIAL").contains(turn.status()))
result.putAll(protocol.visualRecoveryOverlay(turn.id(),actor)); result.putAll(protocol.visualRecoveryOverlay(turn.id(),actor));
return result; return AnalysisRuntimeEngine.publicMetadata(result);
} }
@Transactional @Transactional
@@ -389,6 +411,9 @@ public class AnalysisTurnService {
jdbc.queryForObject("SELECT id FROM analysis_session WHERE id=? FOR UPDATE", String.class, turn.sessionId()); jdbc.queryForObject("SELECT id FROM analysis_session WHERE id=? FOR UPDATE", String.class, turn.sessionId());
turn = requireTurn(turnId, actor); turn = requireTurn(turnId, actor);
requireWritableSession(turn.sessionId()); requireWritableSession(turn.sessionId());
if (protocol != null && !protocol.ownsTurn(turnId)) throw conflict("此轮次缺少有效的协议绑定,请新建会话");
if (dshRunner != null && !AnalysisRuntimeEngine.canExecute(readNullable(turn.planJson(), Map.of())))
throw conflict("此历史轮次使用旧引擎,请重新发送问题以使用 DeepSeek Harness");
if (!List.of("FAILED", "CANCELLED", "PARTIAL").contains(turn.status())) throw conflict("仅未完成的分析任务可以重试"); if (!List.of("FAILED", "CANCELLED", "PARTIAL").contains(turn.status())) throw conflict("仅未完成的分析任务可以重试");
Integer active = jdbc.queryForObject("SELECT COUNT(*) FROM analysis_turn WHERE session_id=? AND status IN ('QUEUED','RUNNING') AND id<>?", Integer active = jdbc.queryForObject("SELECT COUNT(*) FROM analysis_turn WHERE session_id=? AND status IN ('QUEUED','RUNNING') AND id<>?",
Integer.class, turn.sessionId(), turnId); Integer.class, turn.sessionId(), turnId);
@@ -402,7 +427,7 @@ public class AnalysisTurnService {
return Map.of("turn",turnEnvelope(turnId,turn.sessionId(),"")); return Map.of("turn",turnEnvelope(turnId,turn.sessionId(),""));
} }
Map<String, Object> storedPlan = readNullable(turn.planJson(), Map.of()); Map<String, Object> storedPlan = readNullable(turn.planJson(), Map.of());
if (naoRunner != null && isNativePlan(storedPlan)) if (dshRunner != null && isNativePlan(storedPlan))
return retryNative(turn, session, actor, storedPlan); return retryNative(turn, session, actor, storedPlan);
Map<String, Object> storedSemantic = mapValue(storedPlan.get("semantic")); Map<String, Object> storedSemantic = mapValue(storedPlan.get("semantic"));
if (!storedPlan.containsKey("frozenContext") && storedSemantic.isEmpty()) if (!storedPlan.containsKey("frozenContext") && storedSemantic.isEmpty())
@@ -458,8 +483,9 @@ public class AnalysisTurnService {
WHERE id=? AND user_id=? AND status IN ('FAILED','CANCELLED','PARTIAL') AND auth_scope_hash=? WHERE id=? AND user_id=? AND status IN ('FAILED','CANCELLED','PARTIAL') AND auth_scope_hash=?
""", write(retryPlan), write(budget()), turn.id(), actor.getId().toString(), turn.authScopeHash()); """, write(retryPlan), write(budget()), turn.id(), actor.getId().toString(), turn.authScopeHash());
if (updated != 1) throw conflict("任务状态或授权已变化,请刷新后再试"); if (updated != 1) throw conflict("任务状态或授权已变化,请刷新后再试");
appendEvent(turn.id(), "progress", Map.of("turnId", turn.id(), "status", "QUEUED", String retryEngine=AnalysisRuntimeEngine.fromPlan(retryPlan);
"message", "已重新校验原查询条件,将从原装 NAO 重新分析。")); appendEvent(turn.id(), "progress", Map.of("turnId", turn.id(), "status", "QUEUED", "engine",retryEngine,
"message", "已重新校验原查询条件,将从"+AnalysisRuntimeEngine.runtimeName(retryEngine)+"重新分析。"));
submitAfterCommit(turn.id()); submitAfterCommit(turn.id());
return Map.of("turn", turnEnvelope(turn.id(), turn.sessionId(), "")); return Map.of("turn", turnEnvelope(turn.id(), turn.sessionId(), ""));
} }
@@ -486,9 +512,7 @@ public class AnalysisTurnService {
private record NativeTurnOrder(String id, Long firstEventId) {} private record NativeTurnOrder(String id, Long firstEventId) {}
static boolean isNativePlan(Map<String, Object> plan) { static boolean isNativePlan(Map<String, Object> plan) {
return Boolean.TRUE.equals(plan.get("nativeRuntime")) return AnalysisRuntimeEngine.isRuntimePlan(plan);
|| "NAO".equals(plan.get("engine"))
|| "NATIVE_NAO".equals(plan.get("status"));
} }
static Map<String, Object> nativeRetryPlan(Map<String, Object> storedPlan, String fixedReleaseId) { static Map<String, Object> nativeRetryPlan(Map<String, Object> storedPlan, String fixedReleaseId) {
@@ -498,7 +522,7 @@ public class AnalysisTurnService {
static Map<String, Object> nativeRetryPlan(Map<String, Object> storedPlan, String fixedReleaseId, Long eventFloor) { static Map<String, Object> nativeRetryPlan(Map<String, Object> storedPlan, String fixedReleaseId, Long eventFloor) {
Map<String, Object> source = stringMapOrEmpty(storedPlan.get("frozenContext")); Map<String, Object> source = stringMapOrEmpty(storedPlan.get("frozenContext"));
if (source.isEmpty()) source = stringMapOrEmpty(storedPlan.get("context")); if (source.isEmpty()) source = stringMapOrEmpty(storedPlan.get("context"));
if (source.isEmpty()) throw new IllegalArgumentException("原装 NAO 任务未保存原始查询条件,不能安全重试"); if (source.isEmpty()) throw new IllegalArgumentException("原分析任务未保存原始查询条件,不能安全重试");
String planRelease = text(storedPlan.get("releaseId")); String planRelease = text(storedPlan.get("releaseId"));
String contextRelease = text(source.get("releaseId")); String contextRelease = text(source.get("releaseId"));
@@ -521,9 +545,10 @@ public class AnalysisTurnService {
context.put("releaseId", fixedReleaseId); context.put("releaseId", fixedReleaseId);
Map<String, Object> plan = new LinkedHashMap<>(); Map<String, Object> plan = new LinkedHashMap<>();
plan.put("status", "NATIVE_NAO_PENDING"); String engine = AnalysisRuntimeEngine.require(storedPlan.get("engine"));
plan.put("status", AnalysisRuntimeEngine.pendingStatus(engine));
plan.put("nativeRuntime", true); plan.put("nativeRuntime", true);
plan.put("engine", "NAO"); plan.put("engine", engine);
plan.put("releaseId", fixedReleaseId); plan.put("releaseId", fixedReleaseId);
plan.put("context", context); plan.put("context", context);
plan.put("frozenContext", context); plan.put("frozenContext", context);
@@ -560,14 +585,15 @@ public class AnalysisTurnService {
Turn turn = requireTurn(turnId, actor); Turn turn = requireTurn(turnId, actor);
if(protocol!=null&&protocol.ownsTurn(turnId)) { if(protocol!=null&&protocol.ownsTurn(turnId)) {
String attemptId=protocol.latest(turnId).attemptId(); String attemptId=protocol.latest(turnId).attemptId();
Map<String,Object> result=protocol.cancel(turnId,actor,cap(clean(reason)==null?"用户请求停止":reason,1000)); Map<String,Object> result=protocol.cancel(turnId,actor,cap(clean(reason)==null?"用户请求停止":reason,1000),
naoAccess.revokeAttempt(attemptId);naoRunner.requestCancel(attemptId);return result; runtimeModelOutputs.getOrDefault(attemptId,Map.of()));
runtimeAccess.revokeAttempt(attemptId);dshRunner.requestCancel(attemptId);return result;
} }
if (!List.of("QUEUED", "RUNNING").contains(turn.status())) return Map.of("turnId", turnId, "status", turn.status(), "cancelled", false); if (!List.of("QUEUED", "RUNNING").contains(turn.status())) return Map.of("turnId", turnId, "status", turn.status(), "cancelled", false);
jdbc.update("UPDATE analysis_turn SET cancel_requested=TRUE,stop_reason=? WHERE id=? AND user_id=? AND status IN ('QUEUED','RUNNING')", jdbc.update("UPDATE analysis_turn SET cancel_requested=TRUE,stop_reason=? WHERE id=? AND user_id=? AND status IN ('QUEUED','RUNNING')",
cap(clean(reason) == null ? "用户请求停止" : reason, 1000), turnId, actor.getId().toString()); cap(clean(reason) == null ? "用户请求停止" : reason, 1000), turnId, actor.getId().toString());
if (naoAccess != null) naoAccess.revokeTurn(turnId); if (runtimeAccess != null) runtimeAccess.revokeTurn(turnId);
if (naoRunner != null) naoRunner.requestCancel(turnId); if (dshRunner != null) dshRunner.requestCancel(turnId);
appendEvent(turnId, "progress", Map.of("turnId", turnId, "status", turn.status(), "message", "已收到停止请求")); appendEvent(turnId, "progress", Map.of("turnId", turnId, "status", turn.status(), "message", "已收到停止请求"));
return Map.of("turnId", turnId, "status", turn.status(), "cancelRequested", true); return Map.of("turnId", turnId, "status", turn.status(), "cancelRequested", true);
} }
@@ -589,41 +615,94 @@ public class AnalysisTurnService {
requireTurn(turnId, actor); requireTurn(turnId, actor);
SseEmitter emitter = new SseEmitter(60_000L); SseEmitter emitter = new SseEmitter(60_000L);
long initial = lastEventId == null || lastEventId < 0 ? 0 : lastEventId; long initial = lastEventId == null || lastEventId < 0 ? 0 : lastEventId;
try { java.util.concurrent.ForkJoinPool.commonPool().execute(() -> stream(turnId, initial, actor.getId(), emitter)); } startStream(turnId, initial, actor.getId(), emitter);
catch (RuntimeException e) { emitter.completeWithError(e); }
return emitter; return emitter;
} }
private void stream(String turnId, long cursor, UUID userId, SseEmitter emitter) { SseEmitter followSession(Map<String, Object> snapshot, AppUserPrincipal actor) {
SseEmitter emitter = new SseEmitter(60_000L);
try { emitter.send(SseEmitter.event().name("snapshot").data(snapshot)); }
catch (java.io.IOException error) { emitter.completeWithError(error); return emitter; }
String turnId = Objects.toString(snapshot.get("activeTurnId"), "");
if (turnId.isBlank()) {
emitter.complete();
} else {
// The history query reads the row and its event cursor in one consistent statement.
// Events committed after that read remain in the durable stream.
long cursor = mapList(snapshot.get("turns")).stream().filter(row -> turnId.equals(row.get("id")))
.map(row -> row.get("eventCursor")).filter(Objects::nonNull)
.mapToLong(value -> Long.parseLong(value.toString())).findFirst().orElse(0L);
startStream(turnId, cursor, actor.getId(), emitter);
}
return emitter;
}
private void startStream(String turnId, long cursor, UUID userId, SseEmitter emitter) {
var closed = new java.util.concurrent.atomic.AtomicBoolean();
emitter.onCompletion(() -> closed.set(true));
emitter.onTimeout(() -> closed.set(true));
emitter.onError(error -> closed.set(true));
Thread.startVirtualThread(() -> stream(turnId, cursor, userId, emitter, closed));
}
private void stream(String turnId, long cursor, UUID userId, SseEmitter emitter,
java.util.concurrent.atomic.AtomicBoolean closed) {
long stopAt = System.currentTimeMillis() + 55_000L; long stopAt = System.currentTimeMillis() + 55_000L;
String priorDraft = "";
String priorAttempt = "";
try { try {
while (System.currentTimeMillis() < stopAt) { while (!closed.get() && System.currentTimeMillis() < stopAt) {
AppUserPrincipal current = currentUser(userId); AppUserPrincipal current = currentUser(userId);
requireAnalysisAccess(current);
Turn turn = requireTurn(turnId, current); Turn turn = requireTurn(turnId, current);
cursor = nativeEventCursor(cursor, readNullable(turn.planJson(), Map.of())); Map<String, Object> plan = readNullable(turn.planJson(), Map.of());
cursor = nativeEventCursor(cursor, plan);
List<Event> events = jdbc.query("SELECT id,event_type,payload_json FROM analysis_turn_event WHERE turn_id=? AND id>? ORDER BY id LIMIT 100", List<Event> events = jdbc.query("SELECT id,event_type,payload_json FROM analysis_turn_event WHERE turn_id=? AND id>? ORDER BY id LIMIT 100",
(rs, n) -> new Event(rs.getLong(1), rs.getString(2), read(rs.getString(3))), turnId, cursor); (rs, n) -> new Event(rs.getLong(1), rs.getString(2), read(rs.getString(3))), turnId, cursor);
boolean sentTerminal = false;
for (Event event : events) { for (Event event : events) {
if (closed.get()) return;
AppUserPrincipal sendAs = currentUser(userId); AppUserPrincipal sendAs = currentUser(userId);
requireAnalysisAccess(sendAs);
Turn sendTurn = requireTurn(turnId, sendAs); Turn sendTurn = requireTurn(turnId, sendAs);
Object payload = event.payload(); Map<String, Object> payload = AnalysisRuntimeEngine.publicMetadata(event.payload());
if (payload instanceof Map<?, ?> map && map.get("result") instanceof Map<?, ?> result) { if (payload.get("result") instanceof Map<?, ?> result) {
Map<String, Object> safe = new LinkedHashMap<>(event.payload()); String status = Objects.toString(payload.get("status"), "");
Map<String, Object> safeResult = stringMap(result); if (Set.of("COMPLETED", "PARTIAL").contains(status) && protocol != null && protocol.ownsTurn(turnId)) {
Object kind = mapValue(readNullable(sendTurn.planJson(), Map.of()).get("task")).get("kind");
if ("report".equals(kind) || AnalysisProtocolInspection.KIND.equals(kind)) {
asSecurityActor(sendAs, () -> {
if ("report".equals(kind)) protocol.authorizeReportSnapshot(turnId, sendAs);
else protocol.authorizeInspectionSnapshot(turnId, sendAs);
return null;
});
}
}
Map<String, Object> safeResult = AnalysisRuntimeEngine.publicMetadata(stringMap(result));
if (safeResult.get("results") instanceof List<?> values) safeResult.put("results", filterResultList(values, sendAs)); if (safeResult.get("results") instanceof List<?> values) safeResult.put("results", filterResultList(values, sendAs));
if (safeResult.containsKey("visualSchemaVersion") || safeResult.containsKey("visualState") || safeResult.containsKey("visuals")) { if (safeResult.containsKey("visualSchemaVersion") || safeResult.containsKey("visualState") || safeResult.containsKey("visuals")) {
safeResult.putAll(visibleVisuals(safeResult, sendTurn, sendAs, queryContext(sendTurn))); safeResult.putAll(visibleVisuals(safeResult, sendTurn, sendAs, queryContext(sendTurn)));
} }
safe.put("result", safeResult); payload.put("result", safeResult);
payload = safe;
} }
emitter.send(SseEmitter.event().id(Long.toString(event.id())).name(event.type()).data(payload)); emitter.send(SseEmitter.event().id(Long.toString(event.id())).name(event.type()).data(payload));
cursor = event.id(); cursor = event.id();
sentTerminal |= terminal(Objects.toString(payload.get("status"), ""));
} }
if (terminal(turn.status()) && events.isEmpty()) { emitter.complete(); return; } if (terminal(turn.status()) && events.isEmpty()) { emitter.complete(); return; }
if (!sentTerminal && "RUNNING".equals(turn.status())) {
String attempt = Objects.toString(plan.get("attemptId"), turnId);
String draft = runtimeDrafts.getOrDefault(attempt, "");
if ((!draft.isEmpty() || !priorDraft.isEmpty())
&& (!attempt.equals(priorAttempt) || !draft.equals(priorDraft))) {
emitter.send(SseEmitter.event().name("draft").data(Map.of("turnId", turnId, "attemptId", attempt, "text", draft)));
priorAttempt = attempt;
priorDraft = draft;
}
}
Thread.sleep(300L); Thread.sleep(300L);
} }
emitter.send(SseEmitter.event().comment("heartbeat")); emitter.complete(); if (!closed.get()) emitter.complete();
} catch (InterruptedException e) { } catch (InterruptedException e) {
Thread.currentThread().interrupt(); emitter.complete(); Thread.currentThread().interrupt(); emitter.complete();
} catch (Exception e) { emitter.completeWithError(e); } } catch (Exception e) { emitter.completeWithError(e); }
@@ -673,21 +752,17 @@ public class AnalysisTurnService {
appendEvent(turnId,"cancelled",Map.of("turnId",turnId,"status","CANCELLED","stopReason",reason)); appendEvent(turnId,"cancelled",Map.of("turnId",turnId,"status","CANCELLED","stopReason",reason));
return; return;
} }
if(protocol!=null&&readNullable(turn.planJson(),Map.of()).containsKey("protocolVersion")) { if (dshRunner != null && !AnalysisRuntimeEngine.canExecute(readNullable(turn.planJson(), Map.of())))
throw conflict("此历史轮次使用旧引擎,请重新发送问题以使用 DeepSeek Harness");
if(protocol!=null&&protocol.ownsTurn(turnId)) {
runProtocolNative(turn,actor,session);return; runProtocolNative(turn,actor,session);return;
} }
if(protocol!=null&&protocol.ownsSession(turn.sessionId()))throw conflict("任务缺少有效执行绑定,请在新会话中重新分析"); if (dshRunner != null || (protocol != null && protocol.ownsSession(turn.sessionId())))
throw conflict("任务缺少有效执行绑定,请在新会话中重新分析");
if (cancelRequested(turnId)) { finishCancelled(turnId); return; } if (cancelRequested(turnId)) { finishCancelled(turnId); return; }
// Native NAO has its own system-policy execution budget, initialized once its worker starts; ensureWithinDeadline(turn);
// do not let the legacy 240-second queue-time deadline expire before it can run.
if (naoRunner == null) ensureWithinDeadline(turn);
jdbc.update("UPDATE analysis_turn SET status='RUNNING',started_at=CURRENT_TIMESTAMP(6) WHERE id=? AND status='QUEUED'", turnId); jdbc.update("UPDATE analysis_turn SET status='RUNNING',started_at=CURRENT_TIMESTAMP(6) WHERE id=? AND status='QUEUED'", turnId);
appendEvent(turnId, "progress", Map.of("turnId", turnId, "status", "RUNNING", "message", appendEvent(turnId, "progress", Map.of("turnId", turnId, "status", "RUNNING", "message", "正在解析受控查询计划", "engine", "CONTROLLED"));
naoRunner == null ? "正在解析受控查询计划" : "正在连接原装 NAO"));
if (naoRunner != null) {
runNative(turn, actor, session);
return;
}
Map<String, Object> persistedPlan = readNullable(turn.planJson(), Map.of()); Map<String, Object> persistedPlan = readNullable(turn.planJson(), Map.of());
Map<String, Object> persistedSemantic = mapValue(persistedPlan.get("semantic")); Map<String, Object> persistedSemantic = mapValue(persistedPlan.get("semantic"));
Map<String, Object> task = new LinkedHashMap<>(mapValue(persistedPlan.get("task"))); Map<String, Object> task = new LinkedHashMap<>(mapValue(persistedPlan.get("task")));
@@ -942,145 +1017,92 @@ public class AnalysisTurnService {
String attemptId=attempt.attemptId(); String attemptId=attempt.attemptId();
try { try {
if(attempt.input().containsKey("controlledInspection")) {asSecurityActor(actor,()->{runProtocolInspection(attempt,actor,session);return null;});return;} if(attempt.input().containsKey("controlledInspection")) {asSecurityActor(actor,()->{runProtocolInspection(attempt,actor,session);return null;});return;}
String engine=AnalysisRuntimeEngine.fromPlan(readNullable(turn.planJson(),Map.of()));
initializeNativeBudget(turn.id(),turn.budgetJson()); initializeNativeBudget(turn.id(),turn.budgetJson());
NaoAnalysisRunner.Result result=naoRunner.run(turn.id(),session.id(),actor,turn.question(),session.releaseId(),protocol.runtimeContext(attempt,actor), Map<String,Object> runtimeContext=new LinkedHashMap<>(protocol.runtimeContext(attempt,actor));
runtimeContext.put("_runtimeEngine",engine);
DshAnalysisRunner.Result result=dshRunner.run(turn.id(),session.id(),actor,turn.question(),session.releaseId(),runtimeContext,
progress->{ progress->{
if(progress.isEmpty()||protocol.isStopped(attemptId))return; if(progress.isEmpty()||protocol.isStopped(attemptId))return;
AppUserPrincipal fresh=currentUser(actor.getId()); AppUserPrincipal fresh=currentUser(actor.getId());
protocol.progress(attemptId,progress,fresh); if(progress.get("dshDraft") instanceof String draft) {
},answer->{ /* Only a verified terminal answer is published in protocol v1. */ }, requireTurn(turn.id(), fresh);
runtimeDrafts.put(attemptId, draft);
rememberModelOutput(attemptId, progress.containsKey("modelAnswer") ? progress : Map.of("modelAnswer", draft));
} else protocol.progress(attemptId,progress,fresh);
},answer->rememberModelOutput(attemptId,Map.of("modelAnswer",answer)),
()->protocol.isStopped(attemptId)); ()->protocol.isStopped(attemptId));
if(protocol.isStopped(attemptId))return; Map<String,Object> output = new LinkedHashMap<>(result.details());
String original = AnalysisProtocolService.modelAnswer(result.details(), Map.of("answer", result.answer()));
if (original.isBlank()) original = AnalysisProtocolService.modelAnswer(runtimeModelOutputs.getOrDefault(attemptId,Map.of()));
output.put("modelAnswer", original);
rememberModelOutput(attemptId, output);
if(protocol.isStopped(attemptId)) {
preserveStoppedModelOutput(attemptId,actor.getId());
return;
}
if("FAILED".equals(result.status())) { if("FAILED".equals(result.status())) {
String reason=result.details().get("message") instanceof String message?message:"本轮分析未完成,请重试"; String reason=result.details().get("message") instanceof String message?message:"本轮分析未完成,请重试";
protocol.fail(attemptId,cap(reason,600));return; protocol.fail(attemptId,cap(reason,600));
preserveStoppedModelOutput(attemptId,actor.getId());
return;
} }
AppUserPrincipal fresh=currentUser(actor.getId()); AppUserPrincipal fresh=currentUser(actor.getId());
var current=protocol.attempt(attemptId); var current=protocol.attempt(attemptId);
Map<String,Object> task=protocol.task(current); Map<String,Object> task=protocol.task(current);
Map<String,Object> context=AnalysisProtocolService.taskContext(session.releaseId(),task); Map<String,Object> context=AnalysisProtocolService.taskContext(session.releaseId(),task);
Map<String,Object> details=new LinkedHashMap<>(result.details()); Map<String,Object> details=new LinkedHashMap<>(output);
if(!Objects.equals(AnalysisProtocolService.canonical(task),AnalysisProtocolService.canonical(details.get("agentTask")))) if(!Objects.equals(AnalysisProtocolService.canonical(task),AnalysisProtocolService.canonical(details.get("agentTask"))))
throw conflict("本次执行返回的任务与后台修订不一致"); throw conflict("本次执行返回的任务与后台修订不一致");
if(Set.of("COMPLETED","PARTIAL").contains(result.status())&&!"report".equals(task.get("kind")) if(Set.of("COMPLETED","PARTIAL").contains(result.status())&&!"report".equals(task.get("kind"))
&&!Objects.equals(AnalysisProtocolService.canonical(context),AnalysisProtocolService.canonical(details.get("agentContext")))) &&!Objects.equals(AnalysisProtocolService.canonical(context),AnalysisProtocolService.canonical(details.get("agentContext"))))
throw conflict("本次执行返回的查询范围与后台修订不一致"); throw conflict("本次执行返回的查询范围与后台修订不一致");
Map<String,Object> answer=nativeAnswer(details,result.answer(),context,session.releaseId(),result.status(),result.stopReason(),turn.id(),fresh); Map<String,Object> answer=nativeAnswer(details,result.answer(),context,session.releaseId(),result.status(),result.stopReason(),turn.id(),fresh,engine);
answer.put("task",task); answer.put("task",task);
asSecurityActor(fresh,()->{protocol.finish(attemptId,result.status(),answer,details,fresh);return null;}); asSecurityActor(fresh,()->{protocol.finish(attemptId,result.status(),answer,details,fresh);return null;});
} catch(InterruptedException interrupted) { } catch(InterruptedException interrupted) {
Thread.currentThread().interrupt();protocol.fail(attemptId,"执行被中断,请重试"); Thread.currentThread().interrupt();protocol.fail(attemptId,"执行被中断,请重试");
preserveStoppedModelOutput(attemptId,actor.getId());
} catch(Exception failure) { } catch(Exception failure) {
LOGGER.warn("Protocol NAO attempt failed safely: attemptId={} exceptionType={}",attemptId,failure.getClass().getName()); LOGGER.warn("Protocol analysis runtime attempt failed safely: attemptId={} exceptionType={}",attemptId,failure.getClass().getName());
String frames=java.util.Arrays.stream(failure.getStackTrace()).limit(6).map(frame->frame.getClassName()+"#"+frame.getMethodName()+":"+frame.getLineNumber()).collect(java.util.stream.Collectors.joining(" <- ")); String frames=java.util.Arrays.stream(failure.getStackTrace()).limit(6).map(frame->frame.getClassName()+"#"+frame.getMethodName()+":"+frame.getLineNumber()).collect(java.util.stream.Collectors.joining(" <- "));
LOGGER.warn("Protocol analysis failed: turnId={} exceptionType={} frames={}",turn.id(),failure.getClass().getName(),frames); LOGGER.warn("Protocol analysis failed: turnId={} exceptionType={} frames={}",turn.id(),failure.getClass().getName(),frames);
protocol.fail(attemptId,failure instanceof ResponseStatusException status&&status.getReason()!=null?cap(status.getReason(),600):"本轮分析未完成,请在当前授权范围内重试"); protocol.fail(attemptId,failure instanceof ResponseStatusException status&&status.getReason()!=null?cap(status.getReason(),600):"本轮分析未完成,请在当前授权范围内重试");
} finally {if(naoAccess!=null)naoAccess.revokeAttempt(attemptId);} preserveStoppedModelOutput(attemptId,actor.getId());
} finally {
runtimeDrafts.remove(attemptId);
runtimeModelOutputs.remove(attemptId);
if(runtimeAccess!=null)runtimeAccess.revokeAttempt(attemptId);
}
} }
private void runNative(Turn turn, AppUserPrincipal actor, Session session) { private void rememberModelOutput(String attemptId, Map<String,Object> output) {
String turnId = turn.id(); String original = AnalysisProtocolService.modelAnswer(output);
Map<String, Object> plan = new LinkedHashMap<>(); if (original.isBlank()) return;
Map<String, Object> submittedPlan = readNullable(turn.planJson(), Map.of()); var saved = new LinkedHashMap<String,Object>();
Map<String, Object> context = nativeContext(submittedPlan, session.releaseId()); saved.put("modelAnswer", original);
initializeNativeBudget(turnId,turn.budgetJson()); for (String field : List.of("reportMarkdown", "reportTitle", "answerOrigin"))
plan.put("status", "NATIVE_NAO"); if (output.get(field) instanceof String text) saved.put(field, text);
plan.put("nativeRuntime", true); if (output.get("answerValidation") instanceof Map<?,?> validation) saved.put("answerValidation",validation);
plan.put("engine", "NAO"); runtimeModelOutputs.put(attemptId, Map.copyOf(saved));
plan.put("releaseId", session.releaseId()); }
plan.put("startedAt", Instant.now().toString());
plan.put("context", context); private void preserveStoppedModelOutput(String attemptId, UUID userId) {
plan.put("frozenContext", context); Map<String,Object> output = runtimeModelOutputs.getOrDefault(attemptId,Map.of());
plan.put("steps", List.of()); if (output.isEmpty()) return;
preserveNativeEventFloor(submittedPlan, plan);
persistState(turnId, "RUNNING", plan, List.of(), List.of(), null);
long[] lastThinkingPersist = {0};
try { try {
Map<String, Object> runtimeContext = nativeRuntimeContext(context, submittedPlan); AppUserPrincipal fresh = currentUser(userId);
assertScope(currentUser(actor.getId()), turn.authScopeHash()); asSecurityActor(fresh, () -> { protocol.preserveStoppedModelOutput(attemptId,output,fresh); return null; });
runtimeContext.put("_serverHistory", nativeServerHistory( } catch (RuntimeException unavailable) {
turn.sessionId(), turn.userId(), turn.id(), session.releaseId(), turn.authScopeHash())); LOGGER.warn("Stopped model output remains unavailable: attemptId={} exceptionType={}",attemptId,unavailable.getClass().getName());
runtimeContext.put("_serverQuestions", nativeServerQuestions(
turn.sessionId(), turn.userId(), turn.id()));
NaoAnalysisRunner.Result result = naoRunner.run(turnId, session.id(), actor, turn.question(),
session.releaseId(), runtimeContext,
progress -> {
if(progress.isEmpty())return;
if(progress.get("thinkingUpdate") instanceof Map<?,?> update) {
var next=AnalysisThinking.safe(update);
if(next.isEmpty())return;
long now=System.nanoTime();
boolean checkpoint=now-lastThinkingPersist[0]>=java.util.concurrent.TimeUnit.SECONDS.toNanos(5)
|| !"running".equals(next.get("status"));
var delta=AnalysisThinking.merge(plan,next,checkpoint);
if(delta.isEmpty())return;
if(checkpoint) {
persistState(turnId,"RUNNING",plan,mapList(plan.get("steps")),List.of(),null);
lastThinkingPersist[0]=now;
}
appendEvent(turnId,"progress",Map.of("turnId",turnId,"status","RUNNING","thinkingDelta",delta));
return;
}
String message = Objects.toString(progress.getOrDefault("message", "正在分析质量数据…"), "正在分析质量数据…");
if (progress.get("steps") instanceof List<?> steps) plan.put("steps", steps);
if (progress.get("timeline") instanceof List<?> timeline) plan.put("timeline", timeline);
plan.put("toolCount", progress.getOrDefault("toolCount", 0));
plan.put("progressMessage", message);
plan.put("updatedAt", Instant.now().toString());
List<Map<String, Object>> steps = mapList(plan.get("steps"));
persistState(turnId, "RUNNING", plan, steps, List.of(), null);
Map<String, Object> event = new LinkedHashMap<>();
event.put("turnId", turnId); event.put("status", "RUNNING");
event.put("message", message); event.put("toolCount", plan.get("toolCount")); event.put("steps", steps);
event.put("startedAt", plan.get("startedAt")); event.put("updatedAt", plan.get("updatedAt"));
if (plan.containsKey("timeline")) event.put("timeline", plan.get("timeline"));
appendEvent(turnId, "progress", event);
},
answer -> persistNativeAnswer(turnId, UUID.fromString(turn.userId()), turn.authScopeHash(),
session.releaseId(), context, answer),
() -> cancelRequested(turnId));
AppUserPrincipal finalActor = currentUser(UUID.fromString(turn.userId()));
assertScope(finalActor, turn.authScopeHash());
String status = cancelRequested(turnId) ? "CANCELLED" : result.status();
String answerText = result.answer();
String stopReason = result.stopReason();
if ("COMPLETED".equals(status) && answerText.isBlank()) {
status = "PARTIAL";
stopReason = "原装 NAO 未提供最终回答";
}
List<Map<String, Object>> steps = nativeSteps(result.details().get("toolCalls"));
plan.put("steps", steps);
plan.put("toolCount", steps.size());
plan.put("stopReason", stopReason);
if (result.details().get("timeline") instanceof List<?>) plan.put("timeline", NaoAnalysisRunner.safeTimeline(result.details().get("timeline")));
Map<String, Object> answer = nativeAnswer(result.details(), answerText, context, session.releaseId(), status, stopReason,
turn.id(), finalActor);
Map<String, Object> answerContext = mapValue(answer.get("context"));
// The mutable plan context is the trusted displayed/effective scope. Keep
// frozenContext untouched so retry still means the originally submitted turn.
plan.put("context", answerContext);
persistContext(session.id(), finalActor, answerContext);
assertScope(currentUser(UUID.fromString(turn.userId())), turn.authScopeHash());
finish(turnId, status, plan, steps, List.of(), stopReason, answer);
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
naoAccess.revokeTurn(turnId);
finishNativeError(turnId, plan, context, session.releaseId(), cancelRequested(turnId) ? "CANCELLED" : "FAILED",
cancelRequested(turnId) ? "用户请求停止" : "NAO runtime 已中断");
} catch (Exception failure) {
// Do not log subprocess output, model text, token material, or raw exception messages.
naoAccess.revokeTurn(turnId);
LOGGER.warn("Native NAO turn failed safely: turnId={} exceptionType={}", turnId, failure.getClass().getName());
finishNativeError(turnId, plan, context, session.releaseId(), cancelRequested(turnId) ? "CANCELLED" : "FAILED",
cancelRequested(turnId) ? "用户请求停止" : "原装 NAO runtime 未能完成本次分析");
} }
} }
void initializeNativeBudget(String turnId,String currentBudgetJson) { void initializeNativeBudget(String turnId,String currentBudgetJson) {
Map<String,Object> nativeBudget=readNullable(currentBudgetJson,Map.of()); Map<String,Object> nativeBudget=readNullable(currentBudgetJson,Map.of());
nativeBudget.put("toolCallsMax",NAO_TOOL_CALLS_MAX); nativeBudget.put("toolCallsMax",RUNTIME_TOOL_CALLS_MAX);
nativeBudget.put("toolCallsUsed",0); nativeBudget.put("toolCallsUsed",0);
nativeBudget.put("toolCallsRemaining",NAO_TOOL_CALLS_MAX); nativeBudget.put("toolCallsRemaining",RUNTIME_TOOL_CALLS_MAX);
nativeBudget.put("planRevisionsMax",0); nativeBudget.put("planRevisionsMax",0);
nativeBudget.put("planRevisionsUsed",0); nativeBudget.put("planRevisionsUsed",0);
nativeBudget.put("maxOutputTokens",AnalysisRuntimePolicy.MAX_OUTPUT_TOKENS); nativeBudget.put("maxOutputTokens",AnalysisRuntimePolicy.MAX_OUTPUT_TOKENS);
@@ -1091,23 +1113,6 @@ public class AnalysisTurnService {
write(nativeBudget),turnId); write(nativeBudget),turnId);
} }
private Map<String, Object> nativeContext(Map<String, Object> persistedPlan, String releaseId) {
Map<String, Object> frozen = mapValue(persistedPlan.get("frozenContext"));
if (frozen.isEmpty()) frozen = mapValue(persistedPlan.get("context"));
Map<String, Object> context = new LinkedHashMap<>();
for (String key : List.of("year", "month", "monthStart", "monthEnd", "quarter", "department", "group", "product", "projectGroup",
"view", "includeUnassigned", "extraFilters", "stage", "category", "defect", "trendGranularity")) {
if (!frozen.containsKey(key)) continue;
Object value = frozen.get(key);
if ("extraFilters".equals(key)) context.put(key, cn.yunsheng.quality.filters.QueryFilterFields.parse(value));
else if (value instanceof String || value instanceof Number || value instanceof Boolean) context.put(key, value);
}
if (frozen.get("comparisonPeriod") != null)
context.put("comparisonPeriod", validatedAgentPeriod(frozen.get("comparisonPeriod")));
context.put("releaseId", releaseId);
return context;
}
static Map<String, Object> nativeAnswerContext(Object rawAgentContext, Map<String, Object> originalContext, static Map<String, Object> nativeAnswerContext(Object rawAgentContext, Map<String, Object> originalContext,
String fixedReleaseId) { String fixedReleaseId) {
Map<String, Object> fallback = sanitizeNativeAnswerContext(originalContext, fixedReleaseId); Map<String, Object> fallback = sanitizeNativeAnswerContext(originalContext, fixedReleaseId);
@@ -1569,34 +1574,31 @@ public class AnalysisTurnService {
return text.substring(0, text.offsetByCodePoints(0, visiblePoints)) + "…"; return text.substring(0, text.offsetByCodePoints(0, visiblePoints)) + "…";
} }
private void persistNativeAnswer(String turnId, UUID userId, String expectedScopeHash, Map<String, Object> nativeAnswer(Map<String, Object> details, String text, Map<String, Object> context,
String releaseId, Map<String, Object> context, String text) { String releaseId, String status, String stopReason, String turnId, AppUserPrincipal actor) {
if (cancelRequested(turnId)) return; return nativeAnswer(details,text,context,releaseId,status,stopReason,turnId,actor,AnalysisRuntimeEngine.DSH);
assertScope(currentUser(userId), expectedScopeHash);
Map<String, Object> answer = new LinkedHashMap<>();
answer.put("answer", text);
answer.put("results", List.of());
answer.put("context", context);
answer.put("releaseId", releaseId);
answer.put("engine", "NAO");
answer.put("businessResultsSentToModel", true);
jdbc.update("UPDATE analysis_turn SET answer_json=CAST(? AS JSON) WHERE id=? AND status='RUNNING'", write(answer), turnId);
appendEvent(turnId, "answer", Map.of("turnId", turnId, "status", "RUNNING", "result", answer));
} }
Map<String, Object> nativeAnswer(Map<String, Object> details, String text, Map<String, Object> context, Map<String, Object> nativeAnswer(Map<String, Object> details, String text, Map<String, Object> context,
String releaseId, String status, String stopReason, String turnId, AppUserPrincipal actor) { String releaseId, String status, String stopReason, String turnId,
AppUserPrincipal actor, String runtimeEngine) {
String engine=AnalysisRuntimeEngine.require(runtimeEngine);
Object runtimeAgentContext = List.of("COMPLETED", "PARTIAL").contains(status) ? details.get("agentContext") : null; Object runtimeAgentContext = List.of("COMPLETED", "PARTIAL").contains(status) ? details.get("agentContext") : null;
Map<String, Object> answerContext = nativeAnswerContext(runtimeAgentContext, context, releaseId); Map<String, Object> answerContext = nativeAnswerContext(runtimeAgentContext, context, releaseId);
Map<String,Object> protocolTask=stringMapOrEmpty(details.get("agentTask")); Map<String,Object> protocolTask=stringMapOrEmpty(details.get("agentTask"));
if(protocolTask.get("sourceFieldLabels") instanceof Map<?,?> labels&&!labels.isEmpty())answerContext.put("columnLabels",labels); if(protocolTask.get("sourceFieldLabels") instanceof Map<?,?> labels&&!labels.isEmpty())answerContext.put("columnLabels",labels);
if(!protocolTask.containsKey("queryPlan")&&protocolTask.get("stages") instanceof List<?> stages&&stages.size()==1&&Set.of("black","bright").contains(stages.getFirst()))answerContext.put("stage",stages.getFirst()); if(!protocolTask.containsKey("queryPlan")&&protocolTask.get("stages") instanceof List<?> stages&&stages.size()==1&&Set.of("black","bright").contains(stages.getFirst()))answerContext.put("stage",stages.getFirst());
Map<String, Object> answer = new LinkedHashMap<>(); Map<String, Object> answer = new LinkedHashMap<>();
answer.put("answer", text); String original = AnalysisProtocolService.modelAnswer(details, Map.of("answer", text));
answer.put("answer", original);
if (!original.isBlank()) answer.put("modelAnswer", original);
if (details.get("answerOrigin") instanceof String origin && origin.matches("[a-z_]{1,80}")) answer.put("answerOrigin",origin);
if (details.get("answerValidation") instanceof Map<?,?> validation && write(validation).length() <= 16000)
answer.put("answerValidation",read(DshAnalysisRunner.publicText(write(validation))));
answer.put("results", List.of()); answer.put("results", List.of());
answer.put("context", answerContext); answer.put("context", answerContext);
answer.put("releaseId", releaseId); answer.put("releaseId", releaseId);
answer.put("engine", "NAO"); answer.put("engine", engine);
answer.put("businessResultsSentToModel", true); answer.put("businessResultsSentToModel", true);
if("NEEDS_CLARIFICATION".equals(status)&&protocolTask.get("clarificationRequest") instanceof Map<?,?>) if("NEEDS_CLARIFICATION".equals(status)&&protocolTask.get("clarificationRequest") instanceof Map<?,?>)
answer.put("clarificationRequest",protocolTask.get("clarificationRequest")); answer.put("clarificationRequest",protocolTask.get("clarificationRequest"));
@@ -1612,12 +1614,17 @@ public class AnalysisTurnService {
allowedFields, filterKeys, turnId, cleanContext(answerContext)); allowedFields, filterKeys, turnId, cleanContext(answerContext));
answer.putAll(snapshot.asMap()); answer.putAll(snapshot.asMap());
} }
for (String key : List.of("reportMarkdown", "reportHtml", "reportTitle", "nativeChatId")) if (details.get("reportMarkdown") instanceof String markdown
if (details.get(key) instanceof String value && value.length() <= 2_000_000) answer.put(key, value); && !markdown.isBlank() && markdown.length() <= 2_000_000 && markdown.equals(original)) {
answer.put("reportMarkdown", markdown);
if (details.get("reportTitle") instanceof String title && !title.isBlank() && title.length() <= 160)
answer.put("reportTitle", title);
}
if (details.get("nativeChatId") instanceof String chatId && chatId.length() <= 256) answer.put("nativeChatId", chatId);
if (details.get("queryEvidenceRefs") instanceof List<?> refs) if (details.get("queryEvidenceRefs") instanceof List<?> refs)
answer.put("queryEvidenceRefs", refs.stream().filter(value -> value instanceof String queryRef answer.put("queryEvidenceRefs", refs.stream().filter(value -> value instanceof String queryRef
&& queryRef.matches("Q[A-F0-9]{8}")).distinct().limit(100).toList()); && queryRef.matches("Q[A-F0-9]{8}")).distinct().limit(100).toList());
if (details.get("timeline") instanceof List<?>) answer.put("timeline", NaoAnalysisRunner.safeTimeline(details.get("timeline"))); if (details.get("timeline") instanceof List<?>) answer.put("timeline", DshAnalysisRunner.safeTimeline(details.get("timeline")));
if(AnalysisProtocolExploration.nativeAgent(protocolTask))answer.put("executionMode","native_agent"); if(AnalysisProtocolExploration.nativeAgent(protocolTask))answer.put("executionMode","native_agent");
var summary=boundedReadingSummary(details.get("readingSummary"),answer.get("queryEvidenceRefs"));if(!summary.isEmpty())answer.put("readingSummary",summary); var summary=boundedReadingSummary(details.get("readingSummary"),answer.get("queryEvidenceRefs"));if(!summary.isEmpty())answer.put("readingSummary",summary);
if(details.get("answerQuality") instanceof Map<?,?> quality) { if(details.get("answerQuality") instanceof Map<?,?> quality) {
@@ -1643,7 +1650,7 @@ public class AnalysisTurnService {
if(details.get("phaseTimingsMs") instanceof Map<?,?> timings) { if(details.get("phaseTimingsMs") instanceof Map<?,?> timings) {
Map<String,Long> safe=new LinkedHashMap<>(); Map<String,Long> safe=new LinkedHashMap<>();
for(String key:List.of("planningAndReview","taskConfirmation","formalQuery","memberQuery","historySnapshot","historyHeadline","synthesis","visuals","nativeStartup","investigation","firstUsefulMs")) for(String key:List.of("planningAndReview","taskConfirmation","formalQuery","memberQuery","historySnapshot","historyHeadline","synthesis","visuals","nativeStartup","investigation","firstUsefulMs"))
if(timings.get(key) instanceof Number n&&n.longValue()>=0&&n.longValue()<=NAO_DEADLINE_SECONDS*1000L)safe.put(key,n.longValue()); if(timings.get(key) instanceof Number n&&n.longValue()>=0&&n.longValue()<=RUNTIME_DEADLINE_SECONDS*1000L)safe.put(key,n.longValue());
answer.put("phaseTimingsMs",safe); answer.put("phaseTimingsMs",safe);
} }
return answer; return answer;
@@ -1686,7 +1693,7 @@ public class AnalysisTurnService {
if (call.get(field) instanceof String text) step.put(field, cap(text, 500)); if (call.get(field) instanceof String text) step.put(field, cap(text, 500));
if (call.get("input") instanceof Map<?, ?> input) { if (call.get("input") instanceof Map<?, ?> input) {
Map<String, Object> safeInput = new LinkedHashMap<>(); Map<String, Object> safeInput = new LinkedHashMap<>();
for (String field : List.of("mode", "scope", "period", "comparisonPeriod", "filters", "view", "stage", "dimension", "defect", "category", "limit", "sort", "action", "offset", "recordFilter")) { for (String field : List.of("mode", "scope", "period", "comparisonPeriod", "filters", "view", "stage", "dimension", "defect", "category", "metric", "limit", "sort", "action", "offset", "recordFilter")) {
Object item = input.get(field); Object item = input.get(field);
if (item instanceof Map<?, ?> nested) { if (item instanceof Map<?, ?> nested) {
Map<String, Object> safeNested = new LinkedHashMap<>(); Map<String, Object> safeNested = new LinkedHashMap<>();
@@ -17,7 +17,7 @@ import org.springframework.web.servlet.mvc.method.annotation.SseEmitter;
/** Durable, cancellable AI turns; model tools are limited to server-side metrics queries. */ /** Durable, cancellable AI turns; model tools are limited to server-side metrics queries. */
@RestController @RestController
@RequestMapping("/api/analysis") @RequestMapping("/api/analysis")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','BUSINESS_ANALYST','QUALITY_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public class AnalysisTurnsController { public class AnalysisTurnsController {
private final AnalysisTurnService turns; private final AnalysisTurnService turns;
public AnalysisTurnsController(AnalysisTurnService turns) { this.turns = turns; } public AnalysisTurnsController(AnalysisTurnService turns) { this.turns = turns; }
@@ -21,7 +21,7 @@ public final class AnalysisVisualSnapshotAccess {
public static final int SCHEMA_VERSION = 1; public static final int SCHEMA_VERSION = 1;
public static final int MAX_SNAPSHOT_BYTES = 256 * 1024; public static final int MAX_SNAPSHOT_BYTES = 256 * 1024;
private static final int MAX_VISUALS = 12; private static final int MAX_VISUALS = 12;
private static final int MAX_CATEGORIES = 36; private static final int MAX_CATEGORIES = 100;
private static final int MAX_SERIES = 1; private static final int MAX_SERIES = 1;
private static final int MAX_TEXT_BYTES = 2048; private static final int MAX_TEXT_BYTES = 2048;
private static final Pattern SAFE_KEY = Pattern.compile("[A-Za-z0-9][A-Za-z0-9_.:-]{0,119}"); private static final Pattern SAFE_KEY = Pattern.compile("[A-Za-z0-9][A-Za-z0-9_.:-]{0,119}");
@@ -135,8 +135,8 @@ public final class AnalysisVisualSnapshotAccess {
Long totalRows = longInteger(ranking.get("totalRows")); Long totalRows = longInteger(ranking.get("totalRows"));
BigDecimal denominator = decimal(ranking.get("denominator")); BigDecimal denominator = decimal(ranking.get("denominator"));
List<?> rows = ranking.get("rows") instanceof List<?> values ? values : null; List<?> rows = ranking.get("rows") instanceof List<?> values ? values : null;
if (totalRows == null || totalRows < 0 || rows == null || rows.size() != Math.min(10L,totalRows) if (totalRows == null || totalRows < 0 || rows == null || (rows.size() != Math.min(100L,totalRows) && rows.size() != Math.min(10L,totalRows) || rows.size() < Math.min((long)limit,totalRows))
|| denominator == null || denominator.signum() <= 0 || limit < 1 || limit > 10) return false; || denominator == null || denominator.signum() <= 0 || limit < 1 || limit > 100) return false;
List<?> categories = chart.get("categories") instanceof List<?> values ? values : null; List<?> categories = chart.get("categories") instanceof List<?> values ? values : null;
List<?> series = chart.get("series") instanceof List<?> values ? values : null; List<?> series = chart.get("series") instanceof List<?> values ? values : null;
if (categories == null || series == null || series.size() != 1) return false; if (categories == null || series == null || series.size() != 1) return false;
@@ -349,7 +349,7 @@ public final class AnalysisVisualSnapshotAccess {
private static List<Map<String, Object>> normalizeCategories(Object raw, String type, MetricSemantics semantics, private static List<Map<String, Object>> normalizeCategories(Object raw, String type, MetricSemantics semantics,
Set<String> requiredFields, boolean productLabels) { Set<String> requiredFields, boolean productLabels) {
if (!(raw instanceof List<?> values) || values.isEmpty() || values.size() > MAX_CATEGORIES if (!(raw instanceof List<?> values) || values.isEmpty() || values.size() > MAX_CATEGORIES
|| Set.of("bar", "column").contains(type) && values.size() > 10) return null; || Set.of("bar", "column").contains(type) && values.size() > 100) return null;
List<Map<String, Object>> result = new ArrayList<>(); List<Map<String, Object>> result = new ArrayList<>();
Set<String> keys = new LinkedHashSet<>(); Set<String> keys = new LinkedHashSet<>();
for (Object value : values) { for (Object value : values) {
@@ -613,7 +613,7 @@ public final class AnalysisVisualSnapshotAccess {
if ("pareto".equals(type) && "all_scope_weight".equals(coverage.get("shareBasis"))) { if ("pareto".equals(type) && "all_scope_weight".equals(coverage.get("shareBasis"))) {
if (!Set.of("black_total_weight","bright_total_weight","black_scrap_weight","bright_scrap_weight","current_scrap_weight").contains(metric) if (!Set.of("black_total_weight","bright_total_weight","black_scrap_weight","bright_scrap_weight","current_scrap_weight").contains(metric)
|| !"kg".equals(unit) || semantics.stage() == null&&!"current_scrap_weight".equals(metric) || comparisonPeriod != null || !"kg".equals(unit) || semantics.stage() == null&&!"current_scrap_weight".equals(metric) || comparisonPeriod != null
|| categories.isEmpty() || categories.size() > 10 || series.size() != 1 || categories.isEmpty() || categories.size() > 100 || series.size() != 1
|| !integerEquals(coverage.get("returnedRows"),categories.size())) return false; || !integerEquals(coverage.get("returnedRows"),categories.size())) return false;
BigDecimal denominator = decimal(coverage.get("denominator")), sum = BigDecimal.ZERO, previous = null; BigDecimal denominator = decimal(coverage.get("denominator")), sum = BigDecimal.ZERO, previous = null;
if (denominator == null || denominator.signum() <= 0) return false; if (denominator == null || denominator.signum() <= 0) return false;
@@ -698,6 +698,9 @@ public final class AnalysisVisualSnapshotAccess {
&& dimension != null && Set.of("department","group","product").contains(dimension)) && dimension != null && Set.of("department","group","product").contains(dimension))
return section == null || section.equals(dimension); return section == null || section.equals(dimension);
boolean grouped = Set.of("bar", "column").contains(type); boolean grouped = Set.of("bar", "column").contains(type);
if (Set.of("black_scrap_rate", "bright_scrap_rate").contains(metric) && grouped
&& dimension != null && Set.of("department", "group", "product").contains(dimension))
return section == null || section.equals(dimension);
boolean sourceMetric = Set.of("source_defect_weight", "source_defect_delta_weight").contains(metric) boolean sourceMetric = Set.of("source_defect_weight", "source_defect_delta_weight").contains(metric)
&& semantics.sourceDefect(); && semantics.sourceDefect();
if (dimension != null && section == null) { if (dimension != null && section == null) {
@@ -14,8 +14,10 @@ public final class AuthorizationFingerprint {
public static String of(AppUserPrincipal user, GovernanceAccessService governance) { public static String of(AppUserPrincipal user, GovernanceAccessService governance) {
Map<String, Object> policy = governance.policy(user.getId().toString()); Map<String, Object> policy = governance.policy(user.getId().toString());
List<String> roles = governance.fingerprintRoles(user);
if (roles == null || roles.isEmpty()) roles = user.getRoleCodes();
List<String> values = List.of( List<String> values = List.of(
String.join(",", user.getRoleCodes().stream().sorted().toList()), String.join(",", roles.stream().sorted().toList()),
String.join(",", user.getDepartments().stream().sorted().toList()), String.join(",", user.getDepartments().stream().sorted().toList()),
String.join(",", user.getGroups().stream().sorted().toList()), String.join(",", user.getGroups().stream().sorted().toList()),
String.join(",", governance.allowedFields(user).stream().sorted().toList()), String.join(",", governance.allowedFields(user).stream().sorted().toList()),
@@ -0,0 +1,79 @@
package cn.yunsheng.quality.analysis;
import cn.yunsheng.quality.governance.GovernanceAccessService;
import cn.yunsheng.quality.identity.AppUserPrincipal;
import cn.yunsheng.quality.metrics.MetricsResponseAccess;
import cn.yunsheng.quality.metrics.MetricsService;
import cn.yunsheng.quality.metrics.MetricsService.MetricStage;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.util.*;
import org.springframework.http.HttpStatus;
import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.security.core.annotation.AuthenticationPrincipal;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.server.ResponseStatusException;
/** Batched source children joined to the official, authorized period/object denominators. */
@RestController
@RequestMapping("/api/analysis/defects/drilldown")
public class DefectDrilldownController {
private final AnalysisFineDefectsService fine;
private final MetricsService metrics;
private final MetricsResponseAccess metricAccess;
private final GovernanceAccessService governance;
public DefectDrilldownController(AnalysisFineDefectsService fine,MetricsService metrics,MetricsResponseAccess metricAccess,GovernanceAccessService governance){this.fine=fine;this.metrics=metrics;this.metricAccess=metricAccess;this.governance=governance;}
@PostMapping
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String,Object> query(@RequestBody Map<String,Object> input,@AuthenticationPrincipal AppUserPrincipal actor) {
governance.requireDetail(actor);governance.requireBoundAction("QUERY","query.detail");
String dimension=String.valueOf(input.getOrDefault("dimension","period"));
String granularity=String.valueOf(input.getOrDefault("trendGranularity","month"));
if(!Set.of("period","overall","department","group","product","projectGroup").contains(dimension))throw bad("细分维度无效");
if(!Set.of("month","quarter","year").contains(granularity))throw bad("趋势粒度无效");
Map<String,Object> scope=new LinkedHashMap<>(input);scope.remove("trendGranularity");scope.remove("view");scope.put("dimension","defect");scope.put("mode","top");scope.put("limit",100);
// Reuse the source parser to reject unknown filters and validate the selected category/stage.
AnalysisFineDefectsService.Request request=AnalysisFineDefectsService.parse(scope,governance.allowedFields(actor),true);
if(request.category()==null)throw bad("必须指定不良分类");
var query=new MetricsService.MetricsQuery(number(scope,"year"),number(scope,"month"),number(scope,"quarter"),request.department(),request.group(),request.product(),request.projectGroup(),dimension.equals("period")?"overall":dimension,request.includeUnassigned(),request.releaseId(),granularity,AnalysisExplorationTools.exclusions(scope.get("exclusions")))
.withProductSearch(request.productSearch()).withExtraFilters(scope.get("extraFilters")).withColumnFilters(scope.get("columnFilters")).withMonthRange(number(scope,"monthStart"),number(scope,"monthEnd")).withStage(request.stage());
var paired=metrics.calculatePairedSnapshot(query,actor,dimension.equals("period"),() -> fine.grouped(scope,actor,dimension,granularity));
// Refresh the visible fields after the snapshot; a concurrent policy revocation cannot release old fields.
governance.requireDetail(actor);governance.requireBoundAction("QUERY","query.detail");
Set<String> currentAllowed=governance.allowedFields(actor);
Map<String,MetricStage> formal=new LinkedHashMap<>();
if(dimension.equals("period")) {
var snapshot=metricAccess.visibleTrend(paired.trend(),actor);
if(!Objects.equals(snapshot.release().get("id"),request.releaseId()))throw bad("查询数据版本不一致");
snapshot.trend().forEach(row->formal.put(row.period(),request.stage().equals("black")?row.black():row.bright()));
} else {
var snapshot=metricAccess.visible(paired.metrics(),actor);
if(!Objects.equals(snapshot.release().get("id"),request.releaseId()))throw bad("查询数据版本不一致");
if(dimension.equals("overall"))formal.put("overall",request.stage().equals("black")?snapshot.summary().black():snapshot.summary().bright());
else snapshot.rows().forEach(row->formal.put(normalizeKey(row.key()),request.stage().equals("black")?row.black():row.bright()));
}
Map<String,Object> raw=paired.source();
if(!Objects.equals(raw.get("releaseId"),request.releaseId()))throw bad("查询数据版本不一致");
Map<?,?> sourceGroups=(Map<?,?>)raw.get("groups");
List<Map<String,Object>> groups=new ArrayList<>();
formal.forEach((key,stage)->{
Map<?,?> source=sourceGroups.get(key) instanceof Map<?,?> m?m:Map.of();
List<?> rows=source.get("rows") instanceof List<?> list?list:List.of();
List<Map<String,Object>> children=new ArrayList<>();
for(Object item:rows)if(item instanceof Map<?,?> child && (currentAllowed.contains("*")||currentAllowed.contains(String.valueOf(child.get("key"))))) {
BigDecimal weight=child.get("weight") instanceof BigDecimal value?value:null;
var value=new LinkedHashMap<String,Object>();value.put("key",child.get("key"));value.put("weight",weight);value.put("rate",ratio(weight,stage.totalWeight()));children.add(value);
}
BigDecimal parentWeight=stage.defects().stream().filter(d->d.name().equals(request.category())).map(MetricsService.DefectRow::weight).filter(Objects::nonNull).findFirst().orElse(null);
List<?> reconciliation=source.get("categoryReconciliation") instanceof List<?> list?list:List.of();
String status=reconciliation.stream().filter(Map.class::isInstance).map(Map.class::cast).filter(m->request.category().equals(m.get("category"))).map(m->String.valueOf(m.get("reconciliationStatus"))).findFirst().orElse("UNAVAILABLE");
Map<String,Object> group=new LinkedHashMap<>();group.put("key",key);group.put("totalWeight",stage.totalWeight());group.put("parentWeight",parentWeight);group.put("parentRate",ratio(parentWeight,stage.totalWeight()));group.put("children",children);group.put("reconciliationStatus",status);groups.add(group);
});
return Map.of("releaseId",request.releaseId(),"stage",request.stage(),"category",request.category(),"dimension",dimension,"granularity",granularity,"fields",((List<?>)raw.get("fields")).stream().filter(item -> item instanceof Map<?,?> field && (currentAllowed.contains("*")||currentAllowed.contains(String.valueOf(field.get("key"))))).toList(),"groups",groups);
}
private static String normalizeKey(String key){return key==null?"__UNASSIGNED__":key;}
private static Integer number(Map<String,Object> input,String key){Object value=input.get(key);if(value==null)return null;if(!(value instanceof Number n)||n.doubleValue()!=n.intValue())throw bad("期间必须为整数");return n.intValue();}
private static BigDecimal ratio(BigDecimal weight,BigDecimal total){return weight==null||total==null||total.signum()<=0?null:weight.divide(total,12,RoundingMode.HALF_UP);}
private static ResponseStatusException bad(String message){return new ResponseStatusException(HttpStatus.BAD_REQUEST,message);}
}
@@ -13,6 +13,7 @@ import java.util.LinkedHashMap;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.Map;
import java.util.Objects; import java.util.Objects;
import java.util.Set;
import java.util.concurrent.BlockingQueue; import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ArrayBlockingQueue; import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentHashMap;
@@ -23,20 +24,20 @@ import org.springframework.stereotype.Component;
import tools.jackson.databind.json.JsonMapper; import tools.jackson.databind.json.JsonMapper;
import jakarta.annotation.PreDestroy; import jakarta.annotation.PreDestroy;
/** Starts the local original NAO bridge and consumes its strict NDJSON protocol. */ /** Starts the delegated DSH runtime and consumes its strict NDJSON protocol. */
@Component @Component
public class NaoAnalysisRunner { public class DshAnalysisRunner {
private static final long WATCHDOG_MILLIS = AnalysisRuntimePolicy.WATCHDOG_TIMEOUT.toMillis(); private static final long WATCHDOG_MILLIS = AnalysisRuntimePolicy.WATCHDOG_TIMEOUT.toMillis();
private static final long CANCEL_GRACE_MILLIS = Duration.ofSeconds(8).toMillis(); private static final long CANCEL_GRACE_MILLIS = Duration.ofSeconds(8).toMillis();
private static final String STDOUT_EOF = "\u0000<NAO_STDOUT_EOF>"; private static final String STDOUT_EOF = "\u0000<ANALYSIS_RUNTIME_STDOUT_EOF>";
private final JsonMapper json; private final JsonMapper json;
private final NaoRuntimeAccessService access; private final AnalysisRuntimeAccessService access;
private final Map<String, Process> active = new ConcurrentHashMap<>(); private final Map<String, Process> active = new ConcurrentHashMap<>();
private final java.util.Set<String> cancellationRequested = ConcurrentHashMap.newKeySet(); private final java.util.Set<String> cancellationRequested = ConcurrentHashMap.newKeySet();
private final java.util.concurrent.Semaphore visualRecoverySlots=new java.util.concurrent.Semaphore(2); private final java.util.concurrent.Semaphore visualRecoverySlots=new java.util.concurrent.Semaphore(2);
public NaoAnalysisRunner(JsonMapper json, NaoRuntimeAccessService access) { public DshAnalysisRunner(JsonMapper json, AnalysisRuntimeAccessService access) {
this.json = json; this.json = json;
this.access = access; this.access = access;
} }
@@ -47,6 +48,9 @@ public class NaoAnalysisRunner {
BooleanSupplier cancellationRequested) throws IOException, InterruptedException { BooleanSupplier cancellationRequested) throws IOException, InterruptedException {
String attemptId=null; String attemptId=null;
if(context!=null&&context.get("_protocol") instanceof Map<?,?> p&&p.get("attemptId") instanceof String id)attemptId=id; if(context!=null&&context.get("_protocol") instanceof Map<?,?> p&&p.get("attemptId") instanceof String id)attemptId=id;
final String engine;
try { engine = AnalysisRuntimeEngine.require(context == null ? null : context.get("_runtimeEngine")); }
catch (IllegalArgumentException invalidEngine) { throw new IOException("Unsupported analysis runtime engine"); }
String processKey=attemptId==null?turnId:attemptId; String processKey=attemptId==null?turnId:attemptId;
String token = attemptId==null?access.issue(actor, releaseId, turnId):access.issueProtocol(actor,releaseId,turnId,attemptId); String token = attemptId==null?access.issue(actor, releaseId, turnId):access.issueProtocol(actor,releaseId,turnId,attemptId);
Process process = null; Process process = null;
@@ -65,11 +69,12 @@ public class NaoAnalysisRunner {
request.put("question", question); request.put("question", question);
request.put("releaseId", releaseId); request.put("releaseId", releaseId);
request.put("context", context == null ? Map.of() : context); request.put("context", context == null ? Map.of() : context);
request.put("runtimeEngine", engine);
request.put("accessToken", token); request.put("accessToken", token);
stdin.write(json.writeValueAsString(request)); stdin.write(json.writeValueAsString(request));
stdin.newLine(); stdin.newLine();
stdin.flush(); stdin.flush();
return readProtocol(processKey, token, process, stdin, onProgress, onAnswer, cancellationRequested,attemptId); return readProtocol(processKey, token, process, stdin, onProgress, onAnswer, cancellationRequested,attemptId,engine);
} }
} finally { } finally {
if(attemptId==null)access.revokeTurn(turnId);else access.revokeAttempt(attemptId); if(attemptId==null)access.revokeTurn(turnId);else access.revokeAttempt(attemptId);
@@ -92,7 +97,7 @@ public class NaoAnalysisRunner {
if(!visualRecoverySlots.tryAcquire())throw new IOException("图表恢复繁忙,请稍后重试"); if(!visualRecoverySlots.tryAcquire())throw new IOException("图表恢复繁忙,请稍后重试");
Process process=null;String key="visual-recovery:"+recoveryId; Process process=null;String key="visual-recovery:"+recoveryId;
try { try {
Path root=projectRoot(),script=root.resolve("scripts/nao_visual_recovery.py"); Path root=projectRoot(),script=analysisScript(root).resolveSibling("quality_visual_recovery.py");
if(!Files.isRegularFile(script)||!Files.isReadable(script))throw new IOException("图表恢复组件不可用"); if(!Files.isRegularFile(script)||!Files.isReadable(script))throw new IOException("图表恢复组件不可用");
process=new ProcessBuilder("python3",script.toString()).directory(root.toFile()).redirectError(ProcessBuilder.Redirect.DISCARD).start(); process=new ProcessBuilder("python3",script.toString()).directory(root.toFile()).redirectError(ProcessBuilder.Redirect.DISCARD).start();
active.put(key,process);Process running=process; active.put(key,process);Process running=process;
@@ -125,10 +130,10 @@ public class NaoAnalysisRunner {
private Result readProtocol(String turnId, String token, Process process, BufferedWriter stdin, private Result readProtocol(String turnId, String token, Process process, BufferedWriter stdin,
Consumer<Map<String, Object>> onProgress, Consumer<String> onAnswer, Consumer<Map<String, Object>> onProgress, Consumer<String> onAnswer,
BooleanSupplier databaseCancellation,String attemptId) throws IOException, InterruptedException { BooleanSupplier databaseCancellation,String attemptId,String engine) throws IOException, InterruptedException {
BlockingQueue<String> lines = new ArrayBlockingQueue<>(128); BlockingQueue<String> lines = new ArrayBlockingQueue<>(128);
Thread stdoutReader = new Thread(() -> pumpStdout(process, lines), "nao-runtime-stdout-" + shortId(turnId)); Thread stdoutReader = new Thread(() -> pumpStdout(process, lines), "quality-analysis-runtime-stdout-" + shortId(turnId));
Thread stderrReader = new Thread(() -> drainStderr(process), "nao-runtime-stderr-" + shortId(turnId)); Thread stderrReader = new Thread(() -> drainStderr(process), "quality-analysis-runtime-stderr-" + shortId(turnId));
stdoutReader.setDaemon(true); stdoutReader.setDaemon(true);
stderrReader.setDaemon(true); stderrReader.setDaemon(true);
stdoutReader.start(); stdoutReader.start();
@@ -159,34 +164,33 @@ public class NaoAnalysisRunner {
cancelAt = System.nanoTime(); cancelAt = System.nanoTime();
} }
} }
if (!cancelSent && System.nanoTime() >= deadline) throw new IOException("NAO runtime watchdog expired"); if (!cancelSent && System.nanoTime() >= deadline) throw new IOException(AnalysisRuntimeEngine.runtimeName(engine) + " watchdog expired");
if (cancelSent && System.nanoTime() - cancelAt > TimeUnit.MILLISECONDS.toNanos(CANCEL_GRACE_MILLIS)) if (cancelSent && System.nanoTime() - cancelAt > TimeUnit.MILLISECONDS.toNanos(CANCEL_GRACE_MILLIS))
throw new IOException("NAO runtime did not stop after cancellation"); throw new IOException(AnalysisRuntimeEngine.runtimeName(engine) + " did not stop after cancellation");
String line = lines.poll(250, TimeUnit.MILLISECONDS); String line = lines.poll(250, TimeUnit.MILLISECONDS);
if (line == null) { if (line == null) {
if (!process.isAlive() && lines.isEmpty()) throw new IOException("NAO runtime exited without a result"); if (!process.isAlive() && lines.isEmpty()) throw new IOException(AnalysisRuntimeEngine.runtimeName(engine) + " exited without a result");
continue; continue;
} }
if (STDOUT_EOF.equals(line)) throw new IOException("NAO runtime closed stdout without a result"); if (STDOUT_EOF.equals(line)) throw new IOException(AnalysisRuntimeEngine.runtimeName(engine) + " closed stdout without a result");
Map<String, Object> message = parseMessage(line); Map<String, Object> message = parseMessage(line);
String type = Objects.toString(message.get("type"), ""); String type = Objects.toString(message.get("type"), "");
switch (type) { switch (type) {
case "progress" -> onProgress.accept(safeProgress(message)); case "progress" -> onProgress.accept(safeProgress(message, engine));
case "answer" -> { case "answer" -> {
lastAnswer = Objects.toString(message.get("answer"), ""); lastAnswer = AnalysisProtocolService.modelAnswer(message);
onAnswer.accept(lastAnswer); onAnswer.accept(lastAnswer);
} }
case "result" -> { case "result" -> {
if (!lastAnswer.isBlank()) lastAnswer = Objects.toString(message.getOrDefault("answer", lastAnswer), lastAnswer); lastAnswer = AnalysisProtocolService.modelAnswer(message, Map.of("answer", lastAnswer));
else lastAnswer = Objects.toString(message.get("answer"), "");
onAnswer.accept(lastAnswer); onAnswer.accept(lastAnswer);
int exitCode = waitAfterResult(process); int exitCode = waitAfterResult(process);
Result result = Result.from(message, lastAnswer, cancelSent || databaseCancellation.getAsBoolean()); Result result = Result.from(message, lastAnswer, cancelSent || databaseCancellation.getAsBoolean(), engine);
if (modelRevoked) return new Result("FAILED", result.answer(), result.details(), "模型服务授权已失效,本轮分析已停止"); if (modelRevoked) return new Result("FAILED", result.answer(), result.details(), "模型服务授权已失效,本轮分析已停止");
return enforceExitStatus(result, exitCode); return enforceExitStatus(result, exitCode, engine);
} }
default -> throw new IOException("NAO runtime emitted an unsupported message type"); default -> throw new IOException(AnalysisRuntimeEngine.runtimeName(engine) + " emitted an unsupported message type");
} }
} }
} }
@@ -209,24 +213,46 @@ public class NaoAnalysisRunner {
} }
private Map<String, Object> parseMessage(String line) throws IOException { private Map<String, Object> parseMessage(String line) throws IOException {
if (line == null || line.isBlank() || line.length() > 2_000_000) throw new IOException("Invalid NAO runtime output"); if (line == null || line.isBlank() || line.length() > 2_000_000) throw new IOException("Invalid analysis runtime output");
try { try {
Map<?, ?> raw = json.readValue(line, Map.class); Map<?, ?> raw = json.readValue(line, Map.class);
Map<String, Object> decoded = new LinkedHashMap<>(); Map<String, Object> decoded = new LinkedHashMap<>();
raw.forEach((key, value) -> decoded.put(String.valueOf(key), value)); raw.forEach((key, value) -> decoded.put(String.valueOf(key), value));
return decoded; return decoded;
} catch (Exception invalid) { } catch (Exception invalid) {
throw new IOException("NAO runtime output was not valid NDJSON", invalid); throw new IOException("Analysis runtime output was not valid NDJSON", invalid);
} }
} }
static Map<String, Object> safeProgress(Map<String, Object> message) { static Map<String, Object> safeProgress(Map<String, Object> message) {
return safeProgress(message, AnalysisRuntimeEngine.DSH);
}
static Map<String, Object> safeProgress(Map<String, Object> message, String runtimeEngine) {
String engine = AnalysisRuntimeEngine.require(runtimeEngine);
if (message.get("modelAnswer") instanceof String original) {
String publicAnswer = publicText(original);
return Map.of("modelAnswer", publicAnswer, "dshDraft",
message.get("dshDraft") instanceof String draft ? cap(publicText(draft),4000) : cap(publicAnswer,4000));
}
if (message.get("dshDraft") instanceof String draft) {
String publicDraft = publicText(draft).replaceAll("[\\p{Cntrl}&&[^\\r\\n\\t]]", "");
return Map.of("dshDraft", cap(publicDraft, 4000));
}
if (message.get("dshActivity") instanceof String activity) {
return switch (activity) {
case "generating" -> Map.of("engine", engine, "dshActivity", activity, "message", "DSH 正在生成分析…");
case "checking" -> Map.of("engine", engine, "dshActivity", activity, "message", "正在整理回答…");
default -> Map.of();
};
}
if (message.containsKey("thinkingUpdate")) { if (message.containsKey("thinkingUpdate")) {
var thinking=AnalysisThinking.safe(message.get("thinkingUpdate")); var thinking=AnalysisThinking.safe(message.get("thinkingUpdate"));
return thinking.isEmpty()?Map.of():Map.of("thinkingUpdate",thinking); return thinking.isEmpty()?Map.of():Map.of("thinkingUpdate",thinking);
} }
Map<String, Object> safe = new LinkedHashMap<>(); Map<String, Object> safe = new LinkedHashMap<>();
safe.put("message", cap(Objects.toString(message.getOrDefault("message", "原装 NAO 正在处理"), "原装 NAO 正在处理"), 500)); safe.put("message", cap(Objects.toString(message.getOrDefault("message", AnalysisRuntimeEngine.progressLabel(engine)), AnalysisRuntimeEngine.progressLabel(engine)), 500));
safe.put("engine", engine);
Object count = message.get("toolCount"); Object count = message.get("toolCount");
if (count instanceof Number number && number.intValue() >= 0 && number.intValue() <= 1000) if (count instanceof Number number && number.intValue() >= 0 && number.intValue() <= 1000)
safe.put("toolCount", number.intValue()); safe.put("toolCount", number.intValue());
@@ -263,8 +289,9 @@ public class NaoAnalysisRunner {
return List.copyOf(safe); return List.copyOf(safe);
} }
private static String publicText(String value) { static String publicText(String value) {
return value.replaceAll("(?i)Bearer\\s+[A-Za-z0-9._~+/-]+", "Bearer [REDACTED]") return value.replaceAll("(?i)quality-turn:[0-9a-f-]{36}", "")
.replaceAll("(?i)Bearer\\s+[A-Za-z0-9._~+/-]+", "Bearer [REDACTED]")
.replaceAll("(?i)(api[_-]?key|password|token|secret|authorization)(\\s*[=::]\\s*)[^\\s,,;;]+", "$1$2[REDACTED]"); .replaceAll("(?i)(api[_-]?key|password|token|secret|authorization)(\\s*[=::]\\s*)[^\\s,,;;]+", "$1$2[REDACTED]");
} }
@@ -283,37 +310,46 @@ public class NaoAnalysisRunner {
} }
static Result enforceExitStatus(Result result, int exitCode) { static Result enforceExitStatus(Result result, int exitCode) {
return enforceExitStatus(result, exitCode, AnalysisRuntimeEngine.DSH);
}
static Result enforceExitStatus(Result result, int exitCode, String engine) {
if (exitCode == 0 || "FAILED".equals(result.status()) || "CANCELLED".equals(result.status())) return result; if (exitCode == 0 || "FAILED".equals(result.status()) || "CANCELLED".equals(result.status())) return result;
return new Result("FAILED", result.answer(), result.details(), "NAO runtime exited abnormally"); return new Result("FAILED", result.answer(), result.details(), AnalysisRuntimeEngine.runtimeName(engine) + " exited abnormally");
}
private Path analysisScript(Path root) {
String configuredScript = firstNonBlank(System.getProperty("quality.analysis.runtime.script"), System.getenv("QUALITY_ANALYSIS_RUNTIME_SCRIPT"));
return (configuredScript == null ? root.resolve("scripts/quality_analysis_runtime.py") : Path.of(configuredScript))
.toAbsolutePath().normalize();
} }
private RuntimeLocation runtimeLocation() throws IOException { private RuntimeLocation runtimeLocation() throws IOException {
Path root=projectRoot(); Path root = projectRoot();
String configuredScript = firstNonBlank(System.getProperty("quality.analysis.nao.script"), System.getenv("QUALITY_ANALYSIS_NAO_SCRIPT")); Path script = analysisScript(root);
Path script = configuredScript == null ? root.resolve("scripts/nao_quality_runtime.py") : Path.of(configuredScript).toAbsolutePath().normalize(); if (!Files.isRegularFile(script) || !Files.isReadable(script)) throw new IOException("DSH runtime script is missing or unreadable");
if (!Files.isRegularFile(script) || !Files.isReadable(script)) throw new IOException("NAO runtime script is missing or unreadable"); return new RuntimeLocation(root, script);
return new RuntimeLocation(root, script.toAbsolutePath().normalize());
} }
private Path projectRoot() throws IOException { private Path projectRoot() throws IOException {
String configuredRoot = firstNonBlank(System.getProperty("quality.project.root"), System.getenv("QUALITY_PROJECT_ROOT")); String configuredRoot = firstNonBlank(System.getProperty("quality.project.root"), System.getenv("QUALITY_PROJECT_ROOT"));
Path root = configuredRoot == null ? findProjectRoot(Path.of("").toAbsolutePath()) : Path.of(configuredRoot).toAbsolutePath().normalize(); Path root = configuredRoot == null ? findProjectRoot(Path.of("").toAbsolutePath()) : Path.of(configuredRoot).toAbsolutePath().normalize();
if (root == null || !Files.isDirectory(root)) throw new IOException("NAO project root was not found"); if (root == null || !Files.isDirectory(root)) throw new IOException("Quality analysis project root was not found");
return root; return root;
} }
private Path findProjectRoot(Path start) { private Path findProjectRoot(Path start) {
Path candidate = start; Path candidate = start;
for (int depth = 0; depth <= 8 && candidate != null; depth++, candidate = candidate.getParent()) { for (int depth = 0; depth <= 8 && candidate != null; depth++, candidate = candidate.getParent()) {
if (Files.isRegularFile(candidate.resolve("scripts/nao_quality_runtime.py"))) return candidate; if (Files.isRegularFile(candidate.resolve("scripts/quality_analysis_runtime.py"))) return candidate;
} }
return null; return null;
} }
private List<String> command(RuntimeLocation location) throws IOException { private List<String> command(RuntimeLocation location) throws IOException {
String override = firstNonBlank(System.getProperty("quality.analysis.nao.command"), System.getenv("QUALITY_ANALYSIS_NAO_COMMAND")); String override = firstNonBlank(System.getProperty("quality.analysis.runtime.command"), System.getenv("QUALITY_ANALYSIS_RUNTIME_COMMAND"));
if (override == null) return List.of("python3", location.script().toString()); if (override == null) return List.of("python3", location.script().toString());
List<String> parsed = tokenizeCommand(override); List<String> parsed = tokenizeCommand(override);
if (parsed.isEmpty()) throw new IOException("Configured NAO command is empty"); if (parsed.isEmpty()) throw new IOException("Configured analysis runtime command is empty");
return parsed; return parsed;
} }
@@ -365,11 +401,21 @@ public class NaoAnalysisRunner {
public record Result(String status, String answer, Map<String, Object> details, String stopReason) { public record Result(String status, String answer, Map<String, Object> details, String stopReason) {
static Result from(Map<String, Object> message, String streamedAnswer, boolean cancellationRequested) { static Result from(Map<String, Object> message, String streamedAnswer, boolean cancellationRequested) {
return from(message, streamedAnswer, cancellationRequested, AnalysisRuntimeEngine.DSH);
}
static Result from(Map<String, Object> message, String streamedAnswer, boolean cancellationRequested, String selectedEngine) {
String engine = AnalysisRuntimeEngine.require(selectedEngine);
Map<String, Object> details = new LinkedHashMap<>(message); Map<String, Object> details = new LinkedHashMap<>(message);
details.remove("type"); details.remove("type");
String status = Objects.toString(message.get("status"), "FAILED").toUpperCase(java.util.Locale.ROOT); String status = Objects.toString(message.get("status"), "FAILED").toUpperCase(java.util.Locale.ROOT);
String reason = Objects.toString(message.getOrDefault("stopReason", ""), ""); String reason = Objects.toString(message.getOrDefault("stopReason", ""), "");
String answer = Objects.toString(message.getOrDefault("answer", streamedAnswer), streamedAnswer); String answer = message.get("modelAnswer") instanceof String original
? original : Objects.toString(message.getOrDefault("answer", streamedAnswer), streamedAnswer);
if (!answer.isBlank()) {
answer = publicText(answer);
details.put("modelAnswer", answer);
}
String safeMessage = "result".equals(message.get("type")) && "FAILED".equals(status) String safeMessage = "result".equals(message.get("type")) && "FAILED".equals(status)
? safeWorkerMessage(message.get("message")) : null; ? safeWorkerMessage(message.get("message")) : null;
if (cancellationRequested || "CANCELLED".equals(status)) { if (cancellationRequested || "CANCELLED".equals(status)) {
@@ -383,7 +429,7 @@ public class NaoAnalysisRunner {
|| !"needs_clarification".equals(reason) || !"needs_clarification".equals(reason)
|| !(message.get("answer") instanceof String prompt) || prompt.isBlank()) { || !(message.get("answer") instanceof String prompt) || prompt.isBlank()) {
status = "FAILED"; status = "FAILED";
reason = "原装 NAO 返回了无效澄清请求"; reason = AnalysisRuntimeEngine.runtimeName(engine) + " 返回了无效澄清请求";
} }
} else if ("COMPLETED".equals(status) && !"stop".equalsIgnoreCase(reason) } else if ("COMPLETED".equals(status) && !"stop".equalsIgnoreCase(reason)
&& !("evidence_finalized".equals(reason) && !("evidence_finalized".equals(reason)
@@ -424,21 +470,41 @@ public class NaoAnalysisRunner {
&& message.get("protocolEvidence") instanceof List<?> proofs&&!proofs.isEmpty() && message.get("protocolEvidence") instanceof List<?> proofs&&!proofs.isEmpty()
&& message.get("evidenceCompletion") instanceof Map<?,?> coverage&&Boolean.TRUE.equals(coverage.get("serverCoverageSufficient")))) { && message.get("evidenceCompletion") instanceof Map<?,?> coverage&&Boolean.TRUE.equals(coverage.get("serverCoverageSufficient")))) {
status = "PARTIAL"; status = "PARTIAL";
if (reason.isBlank()) reason = "原装 NAO 未以 stop 原因正常结束"; if (reason.isBlank()) reason = AnalysisRuntimeEngine.runtimeName(engine) + " 未以 stop 原因正常结束";
} else if (!List.of("COMPLETED", "PARTIAL", "FAILED").contains(status)) { } else if (!List.of("COMPLETED", "PARTIAL", "FAILED").contains(status)) {
status = "FAILED"; status = "FAILED";
if (reason.isBlank()) reason = "原装 NAO 返回了无效结束状态"; if (reason.isBlank()) reason = AnalysisRuntimeEngine.runtimeName(engine) + " 返回了无效结束状态";
}
boolean conflictingExit=Boolean.FALSE.equals(message.get("nativeStopSucceeded"))
|| message.get("finishReason") instanceof String finish&&!finish.isBlank()&&!"stop".equalsIgnoreCase(finish)
|| message.get("toolCalls") instanceof List<?> calls&&calls.stream().anyMatch(raw->raw instanceof Map<?,?> call&&"started".equals(call.get("status")))
|| message.get("answerQuality") instanceof Map<?,?> quality&&Boolean.FALSE.equals(quality.get("narrativeComplete"));
if(Set.of("COMPLETED","PARTIAL").contains(status)&&conflictingExit) {
if("COMPLETED".equals(status)){status="PARTIAL";reason="incomplete_finish";}
var quality=new LinkedHashMap<String,Object>();
if(message.get("answerQuality") instanceof Map<?,?> prior)prior.forEach((k,v)->quality.put(String.valueOf(k),v));
quality.put("version",1);quality.put("narrativeComplete",false);details.put("answerQuality",quality);
}
boolean visibleBody=!answer.isBlank()
|| message.get("reportMarkdown") instanceof String markdown&&!markdown.isBlank()
|| message.get("reportHtml") instanceof String html&&!html.isBlank()
|| message.get("memberList") instanceof Map<?,?> || message.get("recordList") instanceof Map<?,?>;
if("COMPLETED".equals(status)&&!visibleBody) {
status="PARTIAL";reason="answer_missing";
var quality=new LinkedHashMap<String,Object>();
if(message.get("answerQuality") instanceof Map<?,?> prior)prior.forEach((k,v)->quality.put(String.valueOf(k),v));
quality.put("version",1);quality.put("narrativeComplete",false);details.put("answerQuality",quality);
} }
if ("FAILED".equals(status) && safeMessage != null) { if ("FAILED".equals(status) && safeMessage != null) {
details.put("message", safeMessage); details.put("message", safeMessage);
if (answer.isBlank()) answer = safeMessage;
if (reason.isBlank() || "runtime_error".equals(reason) || "invalid_request".equals(reason)) reason = safeMessage; if (reason.isBlank() || "runtime_error".equals(reason) || "invalid_request".equals(reason)) reason = safeMessage;
} else { } else {
details.remove("message"); details.remove("message");
} }
details.put("status", status); details.put("status", status);
details.put("stopReason", reason);
details.put("answer", answer); details.put("answer", answer);
details.put("engine", "NAO"); details.put("engine", engine);
return new Result(status, answer, details, reason); return new Result(status, answer, details, reason);
} }
@@ -0,0 +1,107 @@
package cn.yunsheng.quality.analysis;
import static cn.yunsheng.quality.analysis.AnalysisFineDefectsService.*;
import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.function.Supplier;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.RowCallbackHandler;
import tools.jackson.databind.json.JsonMapper;
/** Category payloads retain the exact counters and original authorization dimensions. */
final class FineDefectCompactRollups {
private static final int FORMAT = 1;
private final JdbcTemplate jdbc;
private final JsonMapper json;
FineDefectCompactRollups(JdbcTemplate jdbc, JsonMapper json) { this.jdbc=jdbc; this.json=json; }
void invalidate(String build) {
jdbc.update("DELETE FROM fine_defect_compact_build WHERE build_id=?",build);
}
void refreshBuild(String build) {
invalidate(build);
jdbc.update("DELETE FROM fine_defect_compact WHERE build_id=?",build);
Integer ready=jdbc.queryForObject("SELECT COUNT(*) FROM fine_defect_rollup_build WHERE build_id=?",Integer.class,build);
if(ready==null||ready!=1)return;
insert(build,"",List.of());
jdbc.update("INSERT INTO fine_defect_compact_build(build_id,format_version) VALUES(?,?)",build,FORMAT);
}
void refreshGroups(String build,String stage,String predicate,List<Object> scopeArgs) {
List<Object> args=new ArrayList<>(List.of(build,stage));args.addAll(scopeArgs);
jdbc.update("DELETE f FROM fine_defect_compact f WHERE f.build_id=? AND f.stage=? AND ("+predicate+")",args.toArray());
List<Object> filters=new ArrayList<>(List.of(stage));filters.addAll(scopeArgs);
insert(build," AND f.stage=? AND ("+predicate+")",filters);
// A partial update cannot make a previously incomplete build ready.
}
private void insert(String build,String predicate,List<Object> scopeArgs) {
List<Object> args=new ArrayList<>(List.of(build));args.addAll(scopeArgs);
String category=categoryExpression();
jdbc.update("""
INSERT INTO fine_defect_compact(build_id,stage,month_key,department,group_name,project_group,
product,category,source_row_count,stats,payload_bytes)
SELECT build_id,stage,month_key,department,group_name,project_group,product,category,
source_row_count,stats,OCTET_LENGTH(CAST(stats AS CHAR)) FROM (
SELECT f.build_id,f.stage,ANY_VALUE(f.month_key) AS month_key,
ANY_VALUE(f.department) AS department,ANY_VALUE(f.group_name) AS group_name,
ANY_VALUE(f.project_group) AS project_group,ANY_VALUE(f.product) AS product,
"""+category+" AS category,"+"""
MAX(f.source_row_count) AS source_row_count,
JSON_ARRAYAGG(JSON_ARRAY(f.stat_kind,f.field_key,CAST(f.weight_sum AS CHAR),
f.present_rows,f.missing_rows,f.invalid_rows,f.negative_rows)) AS stats
FROM fine_defect_rollup f WHERE f.build_id=? AND f.grain='detail'
"""+predicate+"""
GROUP BY f.build_id,f.stage,CAST(f.month_key AS BINARY),CAST(f.department AS BINARY),
CAST(f.group_name AS BINARY),CAST(f.project_group AS BINARY),CAST(f.product AS BINARY),category
) packed WHERE category IS NOT NULL
""",args.toArray());
}
private static String categoryExpression() {
StringBuilder source=new StringBuilder("CASE f.field_key"),formal=new StringBuilder("CASE f.field_key");
for(String stage:List.of("black","bright")) {
for(var field:fields(stage))source.append(" WHEN ").append(literal(field.key())).append(" THEN ").append(literal(field.category()));
for(var category:categories(stage))if(category.available())formal.append(" WHEN ").append(literal(category.componentCode())).append(" THEN ").append(literal(category.name()));
}
return "CASE f.stat_kind WHEN 'source' THEN "+source+" END WHEN 'formal' THEN "+formal+" END WHEN 'expected' THEN f.field_key END";
}
private static String literal(String value) { return "'"+value.replace("'","''")+"'"; }
boolean read(Map<String,Aggregator> groups,Supplier<Aggregator> create,Request request,Set<String> allowed,
String predicate,List<Object> scopeArgs,String bucket) {
Integer missing=jdbc.queryForObject("""
SELECT COUNT(*) FROM release_member rm
LEFT JOIN fine_defect_compact_build cb ON cb.build_id=rm.transform_build_id AND cb.format_version=?
WHERE rm.release_id=? AND cb.build_id IS NULL
""",Integer.class,FORMAT,scopeArgs.getFirst());
if(missing==null||missing>0)return false;
List<Object> args=new ArrayList<>(scopeArgs);args.add(request.stage());args.add(request.category());
String from=" FROM release_member rm JOIN transform_build b ON b.id=rm.transform_build_id AND b.source_revision_id=rm.source_revision_id"
+" JOIN source_revision sr ON sr.id=rm.source_revision_id JOIN fine_defect_compact f ON f.build_id=rm.transform_build_id "
+predicate+" AND f.stage=? AND f.category=?";
// Bound each JSON result before aggregation. Oversized buckets use the existing tall-stat query.
Integer oversized=jdbc.queryForObject("SELECT COUNT(*) FROM (SELECT "+bucket+" AS bucket_key"+from
+" GROUP BY bucket_key HAVING SUM(f.payload_bytes)+4*COUNT(*)>LEAST(4194304,@@max_allowed_packet DIV 4)) oversized",Integer.class,args.toArray());
if(oversized!=null&&oversized>0)return false;
jdbc.query("SELECT "+bucket+" AS bucket_key,JSON_ARRAYAGG(f.stats) AS parts,SUM(f.source_row_count) AS source_row_count"
+from+" GROUP BY bucket_key",(RowCallbackHandler) rs -> {
Aggregator aggregate=groups.computeIfAbsent(rs.getString("bucket_key"),key -> create.get());
long rows=rs.getLong("source_row_count");
// SQL groups with the same collation as formal metrics. Decimal strings avoid float conversion.
for(var part:json.readTree(rs.getString("parts")))for(var stat:part) {
aggregate.mergeRollup(new Aggregator.RollupStat(stat.get(0).asString(),stat.get(1).asString(),
new BigDecimal(stat.get(2).asString()),stat.get(3).longValue(),stat.get(4).longValue(),
stat.get(5).longValue(),stat.get(6).longValue(),rows),allowed);
}
},args.toArray());
return true;
}
}
@@ -25,6 +25,15 @@ public class FineDefectRollupBackfill implements ApplicationRunner {
""", (rs, row) -> rs.getString(1)); """, (rs, row) -> rs.getString(1));
long start = System.nanoTime(); long start = System.nanoTime();
for (String build : missing) rollups.refreshBuild(build); for (String build : missing) rollups.refreshBuild(build);
var compactMissing=jdbc.query("""
SELECT DISTINCT b.id FROM current_release c JOIN release_member rm ON rm.release_id=c.release_id
JOIN transform_build b ON b.id=rm.transform_build_id
JOIN fine_defect_rollup_build rb ON rb.build_id=b.id
LEFT JOIN fine_defect_compact_build cb ON cb.build_id=b.id AND cb.format_version=1
WHERE c.dataset_id='quality' AND b.status='COMPLETED' AND cb.build_id IS NULL ORDER BY b.id
""",(rs,row) -> rs.getString(1));
for(String build:compactMissing)rollups.refreshCompactBuild(build);
if(!compactMissing.isEmpty())LOG.info("Compact fine defect backfill: builds={}, elapsedMs={}",compactMissing.size(),(System.nanoTime()-start)/1_000_000L);
if (!missing.isEmpty()) LOG.info("Fine defect rollup backfill: builds={}, elapsedMs={}", missing.size(), (System.nanoTime() - start) / 1_000_000L); if (!missing.isEmpty()) LOG.info("Fine defect rollup backfill: builds={}, elapsedMs={}", missing.size(), (System.nanoTime() - start) / 1_000_000L);
} }
} }
@@ -23,11 +23,13 @@ import static cn.yunsheng.quality.analysis.AnalysisFineDefectsService.*;
public class FineDefectRollups { public class FineDefectRollups {
private final JdbcTemplate jdbc; private final JdbcTemplate jdbc;
private final JsonMapper json; private final JsonMapper json;
private final FineDefectCompactRollups compact;
public FineDefectRollups(JdbcTemplate jdbc, JsonMapper json) { this.jdbc = jdbc; this.json = json; } public FineDefectRollups(JdbcTemplate jdbc, JsonMapper json) { this.jdbc = jdbc; this.json = json; this.compact = new FineDefectCompactRollups(jdbc,json); }
@Transactional @Transactional
public void refreshBuild(String build) { public void refreshBuild(String build) {
compact.invalidate(build);
jdbc.update("DELETE FROM fine_defect_rollup_build WHERE build_id=?", build); jdbc.update("DELETE FROM fine_defect_rollup_build WHERE build_id=?", build);
jdbc.update("DELETE FROM fine_defect_rollup WHERE build_id=?", build); jdbc.update("DELETE FROM fine_defect_rollup WHERE build_id=?", build);
long rows = 0; long rows = 0;
@@ -39,8 +41,10 @@ public class FineDefectRollups {
} }
refreshOrganizations(build, "", List.of()); refreshOrganizations(build, "", List.of());
jdbc.update("INSERT INTO fine_defect_rollup_build(build_id,source_row_count) VALUES(?,?)", build, rows); jdbc.update("INSERT INTO fine_defect_rollup_build(build_id,source_row_count) VALUES(?,?)", build, rows);
compact.refreshBuild(build);
} }
@Transactional
public void copyBuild(String parent, String child) { public void copyBuild(String parent, String child) {
Integer ready = jdbc.queryForObject("SELECT COUNT(*) FROM fine_defect_rollup_build WHERE build_id=?", Integer.class, parent); Integer ready = jdbc.queryForObject("SELECT COUNT(*) FROM fine_defect_rollup_build WHERE build_id=?", Integer.class, parent);
if (ready == null || ready != 1) { refreshBuild(child); return; } if (ready == null || ready != 1) { refreshBuild(child); return; }
@@ -52,8 +56,10 @@ public class FineDefectRollups {
FROM fine_defect_rollup WHERE build_id=? FROM fine_defect_rollup WHERE build_id=?
""", child, parent); """, child, parent);
jdbc.update("INSERT INTO fine_defect_rollup_build(build_id,source_row_count) SELECT ?,source_row_count FROM fine_defect_rollup_build WHERE build_id=?", child, parent); jdbc.update("INSERT INTO fine_defect_rollup_build(build_id,source_row_count) SELECT ?,source_row_count FROM fine_defect_rollup_build WHERE build_id=?", child, parent);
compact.refreshBuild(child);
} }
@Transactional
public void refreshGroups(String build, Set<GroupKey> keys) { public void refreshGroups(String build, Set<GroupKey> keys) {
for (String stage : List.of("black", "bright")) { for (String stage : List.of("black", "bright")) {
var selected = keys.stream().filter(k -> stage.equals(k.stage())).toList(); var selected = keys.stream().filter(k -> stage.equals(k.stage())).toList();
@@ -65,8 +71,9 @@ public class FineDefectRollups {
List<Object> deletedArgs = new ArrayList<>(List.of(build, stage)); deletedArgs.addAll(args); List<Object> deletedArgs = new ArrayList<>(List.of(build, stage)); deletedArgs.addAll(args);
jdbc.update("DELETE f FROM fine_defect_rollup f WHERE f.build_id=? AND f.stage=? AND f.grain='detail' AND (" + predicate + ")", deletedArgs.toArray()); jdbc.update("DELETE f FROM fine_defect_rollup f WHERE f.build_id=? AND f.stage=? AND f.grain='detail' AND (" + predicate + ")", deletedArgs.toArray());
var groups = aggregate(build, stage, " AND (" + predicate + ")", args); var groups = aggregate(build, stage, " AND (" + predicate + ")", args);
if (!representable(groups)) { jdbc.update("DELETE FROM fine_defect_rollup_build WHERE build_id=?", build); return; } if (!representable(groups)) { jdbc.update("DELETE FROM fine_defect_rollup_build WHERE build_id=?", build); compact.invalidate(build); return; }
insert(build, groups); insert(build, groups);
compact.refreshGroups(build,stage,predicate,args);
} }
var organizations = keys.stream().map(k -> new GroupKey(k.stage(), k.month(), k.department(), k.group(), null, null)).distinct().toList(); var organizations = keys.stream().map(k -> new GroupKey(k.stage(), k.month(), k.department(), k.group(), null, null)).distinct().toList();
if (!organizations.isEmpty()) { if (!organizations.isEmpty()) {
@@ -78,6 +85,9 @@ public class FineDefectRollups {
} }
} }
@Transactional
public void refreshCompactBuild(String build) { compact.refreshBuild(build); }
private void refreshOrganizations(String build, String predicate, List<Object> scopeArgs) { private void refreshOrganizations(String build, String predicate, List<Object> scopeArgs) {
List<Object> args = new ArrayList<>(List.of(build)); args.addAll(scopeArgs); List<Object> args = new ArrayList<>(List.of(build)); args.addAll(scopeArgs);
jdbc.update("DELETE f FROM fine_defect_rollup f WHERE f.build_id=? AND f.grain='organization'" + predicate, args.toArray()); jdbc.update("DELETE f FROM fine_defect_rollup f WHERE f.build_id=? AND f.grain='organization'" + predicate, args.toArray());
@@ -125,6 +135,21 @@ public class FineDefectRollups {
return true; return true;
} }
boolean readGroups(Map<String,Aggregator> groups, java.util.function.Supplier<Aggregator> create,
Request request, Set<String> allowed, String predicate, List<Object> scopeArgs,
String bucket, String dimension) {
Integer missing=jdbc.queryForObject("SELECT COUNT(*) FROM release_member rm LEFT JOIN fine_defect_rollup_build rb ON rb.build_id=rm.transform_build_id WHERE rm.release_id=? AND rb.build_id IS NULL",Integer.class,scopeArgs.getFirst());
if(missing==null||missing>0)return false;
if(dimension.equals("product")&&compact.read(groups,create,request,allowed,predicate,scopeArgs,bucket))return true;
boolean organization=!predicate.contains("f.product")&&!predicate.contains("f.project_group")&&!List.of("product","projectGroup").contains(dimension);
List<Object> args=new ArrayList<>(scopeArgs);args.add(request.stage());args.add(organization?"organization":"detail");
List<String> selected=new ArrayList<>();
for(var stat:create.get().rollupStats()){selected.add("(f.stat_kind=? AND f.field_key=?)");args.add(stat.kind());args.add(stat.key());}
String sql="SELECT "+bucket+" AS bucket_key,f.stat_kind,f.field_key,SUM(f.weight_sum) AS weight_sum,SUM(f.present_rows) AS present_rows,SUM(f.missing_rows) AS missing_rows,SUM(f.invalid_rows) AS invalid_rows,SUM(f.negative_rows) AS negative_rows,SUM(f.source_row_count) AS source_row_count FROM release_member rm JOIN transform_build b ON b.id=rm.transform_build_id AND b.source_revision_id=rm.source_revision_id JOIN source_revision sr ON sr.id=rm.source_revision_id JOIN fine_defect_rollup f ON f.build_id=rm.transform_build_id "+predicate+" AND f.stage=? AND f.grain=? AND ("+String.join(" OR ",selected)+") GROUP BY bucket_key,f.stat_kind,f.field_key";
jdbc.query(sql,(RowCallbackHandler) rs -> groups.computeIfAbsent(rs.getString("bucket_key"),key -> create.get()).mergeRollup(new Aggregator.RollupStat(rs.getString("stat_kind"),rs.getString("field_key"),rs.getBigDecimal("weight_sum"),rs.getLong("present_rows"),rs.getLong("missing_rows"),rs.getLong("invalid_rows"),rs.getLong("negative_rows"),rs.getLong("source_row_count")),allowed),args.toArray());
return true;
}
private Map<GroupKey, Aggregator> aggregate(String build, String stage, String predicate, List<Object> scopeArgs) { private Map<GroupKey, Aggregator> aggregate(String build, String stage, String predicate, List<Object> scopeArgs) {
Request request = new Request(build, "top", null, null, null, null, null, null, null, null, Request request = new Request(build, "top", null, null, null, null, null, null, null, null,
true, stage, "defect", null, null, 100, Map.of()); true, stage, "defect", null, null, 100, Map.of());
@@ -71,7 +71,7 @@ public class AnalyticsController {
return top(stage, dimension, category, releaseId, year, month, quarter, department, group, product, projectGroup, includeUnassigned, limit, extraFilters, null, actor); return top(stage, dimension, category, releaseId, year, month, quarter, department, group, product, projectGroup, includeUnassigned, limit, extraFilters, null, actor);
} }
@GetMapping("/top") @GetMapping("/top")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> top(@RequestParam String stage, public Map<String, Object> top(@RequestParam String stage,
@RequestParam(defaultValue = "category") String dimension, @RequestParam(defaultValue = "category") String dimension,
@RequestParam(required = false) String category, @RequestParam(required = false) String category,
@@ -111,7 +111,7 @@ public class AnalyticsController {
throw new ResponseStatusException(HttpStatus.FORBIDDEN, "当前字段授权不包含所选报废类别"); throw new ResponseStatusException(HttpStatus.FORBIDDEN, "当前字段授权不包含所选报废类别");
List<Object> args = new ArrayList<>(extra.isEmpty()?List.of(selected, selected):List.of(selected)); List<Object> args = new ArrayList<>(extra.isEmpty()?List.of(selected, selected):List.of(selected));
StringBuilder where = baseWhere(args, year, month, quarter, department, group, product, projectGroup, includeUnassigned, actor); StringBuilder where = baseWhere(args, year, month, quarter, department, group, product, projectGroup, includeUnassigned, actor);
new QueryFilterFields(jdbc,governance).append(where,args,stage,extra,actor); new QueryFilterFields(jdbc,governance).append(where,args,stage,extra,actor,selected);
List<AggregateTop> aggregated; List<AggregateTop> aggregated;
if (dimension.equals("category")) { if (dimension.equals("category")) {
List<Component> selectedComponents = selectedCategory == null ? visibleComponents : List.of(selectedCategory); List<Component> selectedComponents = selectedCategory == null ? visibleComponents : List.of(selectedCategory);
@@ -209,7 +209,7 @@ public class AnalyticsController {
} }
@PostMapping("/compare") @PostMapping("/compare")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> compare(@RequestBody CompareRequest request, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> compare(@RequestBody CompareRequest request, @AuthenticationPrincipal AppUserPrincipal actor) {
requireQueryDetail(actor); requireQueryDetail(actor);
if (request == null || request.current() == null || request.comparison() == null) throw bad("current 与 comparison 两组条件必填"); if (request == null || request.current() == null || request.comparison() == null) throw bad("current 与 comparison 两组条件必填");
@@ -284,7 +284,7 @@ public class AnalyticsController {
/** Per-object impact for a single stage. Contributions are returned only for a complete, exclusive partition. */ /** Per-object impact for a single stage. Contributions are returned only for a complete, exclusive partition. */
@PostMapping("/impact") @PostMapping("/impact")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> impact(@RequestBody ImpactRequest request, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> impact(@RequestBody ImpactRequest request, @AuthenticationPrincipal AppUserPrincipal actor) {
requireQueryDetail(actor); requireQueryDetail(actor);
if (request == null || request.current() == null || request.comparison() == null) throw bad("current 与 comparison 两组条件必填"); if (request == null || request.current() == null || request.comparison() == null) throw bad("current 与 comparison 两组条件必填");
@@ -389,7 +389,7 @@ public class AnalyticsController {
} }
@PostMapping("/structure-decomposition") @PostMapping("/structure-decomposition")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> decompose(@RequestBody DecompositionRequest request, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> decompose(@RequestBody DecompositionRequest request, @AuthenticationPrincipal AppUserPrincipal actor) {
requireQueryDetail(actor); requireQueryDetail(actor);
if (request == null || request.current() == null || request.comparison() == null) throw bad("current 与 comparison 两组条件必填"); if (request == null || request.current() == null || request.comparison() == null) throw bad("current 与 comparison 两组条件必填");
@@ -40,7 +40,7 @@ public class ChecklistsController {
} }
@GetMapping @GetMapping
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> list(@AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> list(@AuthenticationPrincipal AppUserPrincipal actor) {
requireQueryDetail(actor); requireQueryDetail(actor);
List<Map<String, Object>> items = jdbc.query("SELECT id,title,row_version,created_at,updated_at FROM analysis_checklist WHERE owner_user_id=? ORDER BY updated_at DESC", List<Map<String, Object>> items = jdbc.query("SELECT id,title,row_version,created_at,updated_at FROM analysis_checklist WHERE owner_user_id=? ORDER BY updated_at DESC",
@@ -50,7 +50,7 @@ public class ChecklistsController {
} }
@PostMapping @PostMapping
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> create(@RequestBody CreateRequest request, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> create(@RequestBody CreateRequest request, @AuthenticationPrincipal AppUserPrincipal actor) {
requireQueryDetail(actor); requireQueryDetail(actor);
if (request == null || request.title() == null || request.title().isBlank() || request.title().length() > 190) throw bad("核查清单名称无效"); if (request == null || request.title() == null || request.title().isBlank() || request.title().length() > 190) throw bad("核查清单名称无效");
@@ -60,7 +60,7 @@ public class ChecklistsController {
} }
@GetMapping("/{id}") @GetMapping("/{id}")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> open(@PathVariable String id, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> open(@PathVariable String id, @AuthenticationPrincipal AppUserPrincipal actor) {
Checklist checklist = requireOwned(id, actor); Checklist checklist = requireOwned(id, actor);
requireQueryDetail(actor); requireQueryDetail(actor);
@@ -96,7 +96,7 @@ public class ChecklistsController {
} }
@PostMapping("/{id}/items") @PostMapping("/{id}/items")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> addItem(@PathVariable String id, @RequestBody ItemRequest request, public Map<String, Object> addItem(@PathVariable String id, @RequestBody ItemRequest request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
if (request == null) throw bad("核查项内容必填"); if (request == null) throw bad("核查项内容必填");
@@ -139,7 +139,7 @@ public class ChecklistsController {
} }
@DeleteMapping("/{id}/items/{itemId}") @DeleteMapping("/{id}/items/{itemId}")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> removeItem(@PathVariable String id, @PathVariable String itemId, public Map<String, Object> removeItem(@PathVariable String id, @PathVariable String itemId,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
requireQueryDetail(actor); requireQueryDetail(actor);
@@ -24,7 +24,7 @@ import org.springframework.web.server.ResponseStatusException;
/** Recipient ACLs are valid only while both users retain the authorization captured at grant time. */ /** Recipient ACLs are valid only while both users retain the authorization captured at grant time. */
@Service @Service
public class ComparisonViewSharingService { public class ComparisonViewSharingService {
private static final Set<String> VIEW_ROLES = Set.of("SYSTEM_ADMIN", "QUALITY_VIEWER", "BUSINESS_ANALYST"); private static final Set<String> VIEW_ROLES = Set.of("SYSTEM_ADMIN", "DATA_ADMIN", "USER");
private final JdbcTemplate jdbc; private final JdbcTemplate jdbc;
private final GovernanceAccessService governance; private final GovernanceAccessService governance;
@@ -49,7 +49,7 @@ public class ComparisonViewSharingService {
SELECT DISTINCT u.id, u.username, u.display_name SELECT DISTINCT u.id, u.username, u.display_name
FROM app_user u JOIN app_user_role r ON r.user_id=u.id FROM app_user u JOIN app_user_role r ON r.user_id=u.id
WHERE u.active=TRUE AND u.id<>? WHERE u.active=TRUE AND u.id<>?
AND r.role_code IN ('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST') AND r.role_code IN ('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')
ORDER BY u.display_name, u.id ORDER BY u.display_name, u.id
""", (rs, n) -> new Candidate(rs.getString("id"), rs.getString("username"), rs.getString("display_name")), """, (rs, n) -> new Candidate(rs.getString("id"), rs.getString("username"), rs.getString("display_name")),
owner.getId().toString()); owner.getId().toString());
@@ -48,7 +48,7 @@ public class ComparisonViewsController {
} }
@GetMapping @GetMapping
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> list(@AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> list(@AuthenticationPrincipal AppUserPrincipal actor) {
actor = sharing.currentAuthorized(actor); actor = sharing.currentAuthorized(actor);
requireQueryDetail(actor); requireQueryDetail(actor);
@@ -64,7 +64,7 @@ public class ComparisonViewsController {
} }
@PostMapping @PostMapping
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> create(@RequestBody ViewRequest request, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> create(@RequestBody ViewRequest request, @AuthenticationPrincipal AppUserPrincipal actor) {
actor = sharing.currentAuthorized(actor); actor = sharing.currentAuthorized(actor);
requireQueryDetail(actor); requireQueryDetail(actor);
@@ -76,7 +76,7 @@ public class ComparisonViewsController {
} }
@GetMapping("/{id}") @GetMapping("/{id}")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> get(@PathVariable String id, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> get(@PathVariable String id, @AuthenticationPrincipal AppUserPrincipal actor) {
actor = sharing.currentAuthorized(actor); actor = sharing.currentAuthorized(actor);
requireQueryDetail(actor); requireQueryDetail(actor);
@@ -84,7 +84,7 @@ public class ComparisonViewsController {
} }
@PutMapping("/{id}") @PutMapping("/{id}")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> update(@PathVariable String id, @RequestBody ViewRequest request, public Map<String, Object> update(@PathVariable String id, @RequestBody ViewRequest request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
actor = sharing.currentAuthorized(actor); actor = sharing.currentAuthorized(actor);
@@ -100,7 +100,7 @@ public class ComparisonViewsController {
} }
@DeleteMapping("/{id}") @DeleteMapping("/{id}")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> delete(@PathVariable String id, @RequestParam(required = false) Long expectedRowVersion, public Map<String, Object> delete(@PathVariable String id, @RequestParam(required = false) Long expectedRowVersion,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
actor = sharing.currentAuthorized(actor); actor = sharing.currentAuthorized(actor);
@@ -114,7 +114,7 @@ public class ComparisonViewsController {
} }
@PostMapping("/{id}/open") @PostMapping("/{id}/open")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> open(@PathVariable String id, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> open(@PathVariable String id, @AuthenticationPrincipal AppUserPrincipal actor) {
actor = sharing.currentAuthorized(actor); actor = sharing.currentAuthorized(actor);
requireQueryDetail(actor); requireQueryDetail(actor);
@@ -159,25 +159,25 @@ public class ComparisonViewsController {
} }
@GetMapping("/share-candidates") @GetMapping("/share-candidates")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> shareCandidates(@AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> shareCandidates(@AuthenticationPrincipal AppUserPrincipal actor) {
return Map.of("items", sharing.candidates(actor)); return Map.of("items", sharing.candidates(actor));
} }
@GetMapping("/shared") @GetMapping("/shared")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> sharedViews(@AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> sharedViews(@AuthenticationPrincipal AppUserPrincipal actor) {
return sharing.sharedViews(actor); return sharing.sharedViews(actor);
} }
@GetMapping("/{id}/shares") @GetMapping("/{id}/shares")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> shares(@PathVariable String id, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> shares(@PathVariable String id, @AuthenticationPrincipal AppUserPrincipal actor) {
return sharing.sharesForOwner(id, actor); return sharing.sharesForOwner(id, actor);
} }
@PostMapping("/{id}/shares") @PostMapping("/{id}/shares")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> share(@PathVariable String id, @RequestBody ShareRequest request, public Map<String, Object> share(@PathVariable String id, @RequestBody ShareRequest request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
return sharing.share(id, request == null ? null : request.recipientUserId(), return sharing.share(id, request == null ? null : request.recipientUserId(),
@@ -185,7 +185,7 @@ public class ComparisonViewsController {
} }
@DeleteMapping("/{id}/shares/{recipientUserId}") @DeleteMapping("/{id}/shares/{recipientUserId}")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> revokeShare(@PathVariable String id, @PathVariable String recipientUserId, public Map<String, Object> revokeShare(@PathVariable String id, @PathVariable String recipientUserId,
@RequestParam Long expectedRowVersion, @RequestParam Long expectedRowVersion,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
@@ -25,8 +25,8 @@ import tools.jackson.databind.json.JsonMapper;
@RestController @RestController
@RequestMapping("/api/analytics") @RequestMapping("/api/analytics")
public class InspectionController { public class InspectionController {
private static final List<String> STRATEGY_ROLES = List.of("SYSTEM_ADMIN", "SYSTEM_OPERATOR"); private static final List<String> STRATEGY_ROLES = List.of("SYSTEM_ADMIN");
private static final List<String> RESULT_ROLES = List.of("SYSTEM_ADMIN", "SYSTEM_OPERATOR", "QUALITY_VIEWER", "BUSINESS_ANALYST"); private static final List<String> RESULT_ROLES = List.of("SYSTEM_ADMIN", "DATA_ADMIN", "USER");
private final JdbcTemplate jdbc; private final JdbcTemplate jdbc;
private final JsonMapper json; private final JsonMapper json;
private final InspectionService inspections; private final InspectionService inspections;
@@ -37,11 +37,11 @@ public class InspectionController {
} }
@GetMapping("/inspection-strategies") @GetMapping("/inspection-strategies")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN')")
public Map<String, Object> strategies(@AuthenticationPrincipal AppUserPrincipal actor) { return inspections.strategies(actor); } public Map<String, Object> strategies(@AuthenticationPrincipal AppUserPrincipal actor) { return inspections.strategies(actor); }
@PostMapping("/inspection-strategies") @PostMapping("/inspection-strategies")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN')")
public Map<String, Object> create(@RequestBody StrategyRequest request, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> create(@RequestBody StrategyRequest request, @AuthenticationPrincipal AppUserPrincipal actor) {
if (request == null || request.name() == null || request.name().isBlank() || request.name().length() > 190) throw bad("巡检策略名称无效"); if (request == null || request.name() == null || request.name().isBlank() || request.name().length() > 190) throw bad("巡检策略名称无效");
Map<String, Object> config = validate(request.config()); Map<String, Object> config = validate(request.config());
@@ -52,11 +52,11 @@ public class InspectionController {
} }
@GetMapping("/inspection-strategies/{id}") @GetMapping("/inspection-strategies/{id}")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN')")
public Map<String, Object> get(@PathVariable String id, @AuthenticationPrincipal AppUserPrincipal actor) { return getStrategy(id, actor); } public Map<String, Object> get(@PathVariable String id, @AuthenticationPrincipal AppUserPrincipal actor) { return getStrategy(id, actor); }
@PutMapping("/inspection-strategies/{id}") @PutMapping("/inspection-strategies/{id}")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN')")
public Map<String, Object> update(@PathVariable String id, @RequestBody StrategyRequest request, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> update(@PathVariable String id, @RequestBody StrategyRequest request, @AuthenticationPrincipal AppUserPrincipal actor) {
if (request == null || request.name() == null || request.name().isBlank() || request.name().length() > 190) throw bad("巡检策略名称无效"); if (request == null || request.name() == null || request.name().isBlank() || request.name().length() > 190) throw bad("巡检策略名称无效");
if (request.expectedRowVersion() == null) throw bad("缺少 expectedRowVersion"); if (request.expectedRowVersion() == null) throw bad("缺少 expectedRowVersion");
@@ -68,7 +68,7 @@ public class InspectionController {
} }
@GetMapping("/inspection-runs") @GetMapping("/inspection-runs")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> runs(@RequestParam(required = false) String releaseId, public Map<String, Object> runs(@RequestParam(required = false) String releaseId,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
requireResultAccess(actor); requireResultAccess(actor);
@@ -76,7 +76,7 @@ public class InspectionController {
} }
@PostMapping("/inspection-runs/run") @PostMapping("/inspection-runs/run")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> run(@RequestBody RunRequest request, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> run(@RequestBody RunRequest request, @AuthenticationPrincipal AppUserPrincipal actor) {
requireResultAccess(actor); requireResultAccess(actor);
if (request == null || request.releaseId() == null || request.releaseId().isBlank()) throw bad("releaseId 必填"); if (request == null || request.releaseId() == null || request.releaseId().isBlank()) throw bad("releaseId 必填");
@@ -95,7 +95,7 @@ public class InspectionController {
private void requireResultAccess(AppUserPrincipal actor) { private void requireResultAccess(AppUserPrincipal actor) {
if (actor == null || !actor.isEnabled()) throw new ResponseStatusException(HttpStatus.FORBIDDEN, "账号已停用"); if (actor == null || !actor.isEnabled()) throw new ResponseStatusException(HttpStatus.FORBIDDEN, "账号已停用");
governance.requireUserActivePolicy(actor); governance.requireUserActivePolicy(actor);
if (actor.getRoleCodes().stream().noneMatch(role -> role.equals("QUALITY_VIEWER") || role.equals("BUSINESS_ANALYST"))) if (actor.getRoleCodes().stream().noneMatch(RESULT_ROLES::contains))
throw new ResponseStatusException(HttpStatus.FORBIDDEN, "巡检结果需要当前业务读取角色"); throw new ResponseStatusException(HttpStatus.FORBIDDEN, "巡检结果需要当前业务读取角色");
governance.requireDetail(actor); governance.requireDetail(actor);
} }
@@ -43,7 +43,7 @@ import tools.jackson.databind.json.JsonMapper;
/** Deterministic inspection over the trusted metrics service; no row data is sent to a model. */ /** Deterministic inspection over the trusted metrics service; no row data is sent to a model. */
@Service @Service
public class InspectionService { public class InspectionService {
private static final Set<String> BUSINESS_READ_ROLES = Set.of("QUALITY_VIEWER", "BUSINESS_ANALYST"); private static final Set<String> BUSINESS_READ_ROLES = Set.of("SYSTEM_ADMIN", "DATA_ADMIN", "USER");
private final JdbcTemplate jdbc; private final JdbcTemplate jdbc;
private final JsonMapper json; private final JsonMapper json;
private final MetricsService metrics; private final MetricsService metrics;
@@ -31,7 +31,7 @@ public class TargetController {
} }
@GetMapping("/targets") @GetMapping("/targets")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> resolve(@RequestParam(required = false) String releaseId, public Map<String, Object> resolve(@RequestParam(required = false) String releaseId,
@RequestParam(defaultValue = "COMPOSITE_RATE") String metric, @RequestParam(defaultValue = "COMPOSITE_RATE") String metric,
@RequestParam(required = false) Integer year, @RequestParam(required = false) Integer year,
@@ -55,42 +55,42 @@ public class TargetController {
} }
@GetMapping({"/targets/versions", "/versions"}) @GetMapping({"/targets/versions", "/versions"})
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_PUBLISHER','BUSINESS_ANALYST','QUALITY_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> versions() { return targets.listVersions(); } public Map<String, Object> versions() { return targets.listVersions(); }
@GetMapping({"/targets/versions/{id}", "/versions/{id}"}) @GetMapping({"/targets/versions/{id}", "/versions/{id}"})
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_PUBLISHER','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> version(@PathVariable String id) { return targets.version(id); } public Map<String, Object> version(@PathVariable String id) { return targets.version(id); }
@PostMapping("/targets/versions") @PostMapping("/targets/versions")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> create(@RequestBody CreateRequest request, @AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> create(@RequestBody CreateRequest request, @AuthenticationPrincipal AppUserPrincipal actor) {
if (request == null) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "目标版本名称必填"); if (request == null) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "目标版本名称必填");
return targets.create(request.name(), actor); return targets.create(request.name(), actor);
} }
@PutMapping({"/targets/versions/{id}", "/versions/{id}"}) @PutMapping({"/targets/versions/{id}", "/versions/{id}"})
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> save(@PathVariable String id, @RequestBody TargetVersionService.SaveRequest request) { public Map<String, Object> save(@PathVariable String id, @RequestBody TargetVersionService.SaveRequest request) {
if (request == null) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "目标版本内容必填"); if (request == null) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "目标版本内容必填");
return targets.save(id, request); return targets.save(id, request);
} }
@PutMapping("/targets/current") @PutMapping("/targets/current")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> saveCurrent(@RequestBody TargetVersionService.SaveRequest request) { public Map<String, Object> saveCurrent(@RequestBody TargetVersionService.SaveRequest request) {
return targets.saveCurrent(request); return targets.saveCurrent(request);
} }
@PostMapping({"/targets/versions/{id}/validate", "/versions/{id}/validate"}) @PostMapping({"/targets/versions/{id}/validate", "/versions/{id}/validate"})
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> validate(@PathVariable String id, @RequestBody VersionAction request) { public Map<String, Object> validate(@PathVariable String id, @RequestBody VersionAction request) {
if (request == null) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "目标版本行号必填"); if (request == null) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "目标版本行号必填");
return targets.validateVersion(id, request.expectedRowVersion()); return targets.validateVersion(id, request.expectedRowVersion());
} }
@PostMapping({"/targets/versions/{id}/activate", "/versions/{id}/activate"}) @PostMapping({"/targets/versions/{id}/activate", "/versions/{id}/activate"})
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_PUBLISHER','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> activate(@PathVariable String id, @RequestBody VersionAction request, public Map<String, Object> activate(@PathVariable String id, @RequestBody VersionAction request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
if (request == null) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "目标版本行号必填"); if (request == null) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "目标版本行号必填");
@@ -39,6 +39,7 @@ public class WorkspaceStateController {
jdbc.queryForList("SELECT * FROM platform_metadata ORDER BY metadata_key"), jdbc.queryForList("SELECT * FROM platform_metadata ORDER BY metadata_key"),
jdbc.queryForList("SELECT id,row_version FROM target_version WHERE status='ACTIVE' ORDER BY id"), jdbc.queryForList("SELECT id,row_version FROM target_version WHERE status='ACTIVE' ORDER BY id"),
jdbc.queryForList("SELECT * FROM query_filter_field ORDER BY field_key"), jdbc.queryForList("SELECT * FROM query_filter_field ORDER BY field_key"),
jdbc.queryForList("SELECT * FROM sales_order_type_current ORDER BY dataset_id"),
jdbc.queryForList("SELECT id,row_version FROM service_identity ORDER BY id"), jdbc.queryForList("SELECT id,row_version FROM service_identity ORDER BY id"),
jdbc.queryForList("SELECT * FROM service_binding ORDER BY service_type"))); jdbc.queryForList("SELECT * FROM service_binding ORDER BY service_type")));
Map<String, String> result = new LinkedHashMap<>(); Map<String, String> result = new LinkedHashMap<>();
@@ -15,10 +15,10 @@ public class MetricDictionaryController {
@PreAuthorize("isAuthenticated()") @PreAuthorize("isAuthenticated()")
public Map<String, Object> list() { return service.list(); } public Map<String, Object> list() { return service.list(); }
@GetMapping("/migration") @GetMapping("/migration")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','RULE_MAINTAINER','BUSINESS_ANALYST','DATA_PUBLISHER','AUDIT_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> migration() { return service.migration(); } public Map<String, Object> migration() { return service.migration(); }
@PutMapping("/{code}") @PutMapping("/{code}")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN')")
public Map<String, Object> update(@PathVariable String code, @RequestBody MetricDictionaryService.Edit request, public Map<String, Object> update(@PathVariable String code, @RequestBody MetricDictionaryService.Edit request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
return service.update(code, request, actor); return service.update(code, request, actor);
@@ -18,27 +18,29 @@ public class FilterFieldsController {
private final ReleaseService releases; private final ReleaseService releases;
public FilterFieldsController(JdbcTemplate jdbc,GovernanceAccessService governance,ReleaseService releases) { this.jdbc=jdbc;this.governance=governance;this.releases=releases; } public FilterFieldsController(JdbcTemplate jdbc,GovernanceAccessService governance,ReleaseService releases) { this.jdbc=jdbc;this.governance=governance;this.releases=releases; }
@GetMapping @GetMapping
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST','DATA_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public List<QueryFilterFields.Field> list(@RequestParam(defaultValue="false") boolean manage,@AuthenticationPrincipal AppUserPrincipal actor) { public List<QueryFilterFields.Field> list(@RequestParam(defaultValue="false") boolean manage,@AuthenticationPrincipal AppUserPrincipal actor) {
governance.requireUserActivePolicy(actor); governance.requireUserActivePolicy(actor);
if(manage && actor.getRoleCodes().stream().anyMatch(r->Set.of("SYSTEM_ADMIN","DATA_MAINTAINER").contains(r)))return new QueryFilterFields(jdbc,governance).all(); if(manage && actor.getRoleCodes().stream().anyMatch(r->Set.of("SYSTEM_ADMIN", "DATA_ADMIN").contains(r)))return new QueryFilterFields(jdbc,governance).all();
return new QueryFilterFields(jdbc,governance).visible(actor); return new QueryFilterFields(jdbc,governance).visible(actor);
} }
@GetMapping("/options") @GetMapping("/options")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String,Object> options(@RequestParam String key,@RequestParam(required=false) String search,@AuthenticationPrincipal AppUserPrincipal actor) { public Map<String,Object> options(@RequestParam String key,@RequestParam(required=false) String search,@RequestParam(required=false) String releaseId,@RequestParam(required=false) String stage,@AuthenticationPrincipal AppUserPrincipal actor) {
governance.requireUserActivePolicy(actor);governance.requireBoundAction("QUERY","query.aggregate"); governance.requireUserActivePolicy(actor);governance.requireBoundAction("QUERY","query.aggregate");
return new QueryFilterFields(jdbc,governance).options(key,search,releases.resolvePublishedRelease(null),actor); return new QueryFilterFields(jdbc,governance).options(key,search,releases.resolvePublishedRelease(releaseId),stage,actor);
} }
@GetMapping("/columns") @GetMapping("/columns")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,List<String>> columns(@AuthenticationPrincipal AppUserPrincipal actor) { public Map<String,List<String>> columns(@AuthenticationPrincipal AppUserPrincipal actor) {
governance.requireUserActivePolicy(actor); governance.requireUserActivePolicy(actor);
Map<String,List<String>> result=new LinkedHashMap<>(); Map<String,List<String>> result=new LinkedHashMap<>();
String release=releases.resolvePublishedRelease(null); String release=releases.resolvePublishedRelease(null);
SourceFilterIndex index=new SourceFilterIndex(jdbc);
for(String stage:List.of("black","bright")) { for(String stage:List.of("black","bright")) {
// Raw source headers are stable within each build. Read a representative row of every current build. // Raw source headers are stable within each build. Read a representative row of every current build.
List<String> raw=jdbc.query("SELECT CAST(JSON_KEYS(f.raw_values) AS CHAR) FROM release_member rm JOIN fact_"+stage+" f ON f.id=(SELECT MIN(f2.id) FROM fact_"+stage+" f2 WHERE f2.build_id=rm.transform_build_id) WHERE rm.release_id=?",(rs,n)->rs.getString(1),release); String hint=index.available()?" FORCE INDEX (ix_"+stage+"_source_catalog)":"";
List<String> raw=jdbc.query("SELECT CAST(JSON_KEYS(f.raw_values) AS CHAR) FROM release_member rm JOIN fact_"+stage+" f ON f.id=(SELECT MIN(f2.id) FROM fact_"+stage+" f2"+hint+" WHERE f2.build_id=rm.transform_build_id) WHERE rm.release_id=?",(rs,n)->rs.getString(1),release);
Set<String> columns=new TreeSet<>(); Set<String> columns=new TreeSet<>();
var json=tools.jackson.databind.json.JsonMapper.builder().build(); var json=tools.jackson.databind.json.JsonMapper.builder().build();
Set<String> allowed=governance.allowedFields(actor); Set<String> allowed=governance.allowedFields(actor);
@@ -49,13 +51,14 @@ public class FilterFieldsController {
} }
@PutMapping @PutMapping
@Transactional @Transactional
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public List<QueryFilterFields.Field> save(@RequestBody List<QueryFilterFields.Field> fields,@AuthenticationPrincipal AppUserPrincipal actor) { public List<QueryFilterFields.Field> save(@RequestBody List<QueryFilterFields.Field> fields,@AuthenticationPrincipal AppUserPrincipal actor) {
governance.requireUserActivePolicy(actor); governance.requireUserActivePolicy(actor);
if(fields==null||fields.size()>20)throw QueryFilterFields.bad("最多配置20个筛选字段"); if(fields==null||fields.size()>20)throw QueryFilterFields.bad("最多配置20个筛选字段");
Set<String> keys=new HashSet<>(),labels=new HashSet<>(); Set<String> keys=new HashSet<>(),labels=new HashSet<>();
Set<String> allowed=governance.allowedFields(actor); Set<String> allowed=governance.allowedFields(actor);
for(var f:fields) { for(var f:fields) {
if(SalesOrderTypes.KEY.equals(f.key())||SalesOrderTypes.EXCLUDE_KEY.equals(f.key()))throw QueryFilterFields.bad("销售订单类型由对应表维护,请勿配置为原始列");
if(f.key()==null||!f.key().matches("[a-zA-Z][a-zA-Z0-9_]{0,63}")||!keys.add(f.key()))throw QueryFilterFields.bad("字段标识无效或重复"); if(f.key()==null||!f.key().matches("[a-zA-Z][a-zA-Z0-9_]{0,63}")||!keys.add(f.key()))throw QueryFilterFields.bad("字段标识无效或重复");
if(f.label()==null||f.label().isBlank()||f.label().length()>80||!labels.add(f.label()))throw QueryFilterFields.bad("名称为空、过长或重复"); if(f.label()==null||f.label().isBlank()||f.label().length()>80||!labels.add(f.label()))throw QueryFilterFields.bad("名称为空、过长或重复");
for(String column:List.of(f.blackColumn()==null?"":f.blackColumn(),f.brightColumn()==null?"":f.brightColumn())) { for(String column:List.of(f.blackColumn()==null?"":f.blackColumn(),f.brightColumn()==null?"":f.brightColumn())) {
@@ -23,7 +23,10 @@ public class QueryFilterFields {
} }
public List<Field> visible(AppUserPrincipal actor) { public List<Field> visible(AppUserPrincipal actor) {
Set<String> allowed=governance.allowedFields(actor); Set<String> allowed=governance.allowedFields(actor);
return all().stream().filter(Field::enabled).filter(f->allowed.contains("*") || (allowed.contains(f.blackColumn()) && allowed.contains(f.brightColumn()))).toList(); List<Field> fields=new ArrayList<>(all());
fields.removeIf(f->SalesOrderTypes.KEY.equals(f.key()));
fields.add(new Field(SalesOrderTypes.KEY,SalesOrderTypes.LABEL,"销售订单号","销售订单",true,true,-1));
return fields.stream().filter(Field::enabled).filter(f->allowed.contains("*") || (allowed.contains(f.blackColumn()) && allowed.contains(f.brightColumn()))).toList();
} }
public static Map<String,List<String>> parse(Object raw) { public static Map<String,List<String>> parse(Object raw) {
if(raw==null || "".equals(raw))return Map.of(); if(raw==null || "".equals(raw))return Map.of();
@@ -45,14 +48,23 @@ public class QueryFilterFields {
return Collections.unmodifiableMap(result); return Collections.unmodifiableMap(result);
} }
public void append(StringBuilder where,List<Object> args,String stage,Map<String,List<String>> filters,AppUserPrincipal actor) { public void append(StringBuilder where,List<Object> args,String stage,Map<String,List<String>> filters,AppUserPrincipal actor) {
append(where,args,stage,filters,actor,null);
}
public void append(StringBuilder where,List<Object> args,String stage,Map<String,List<String>> filters,AppUserPrincipal actor,String releaseId) {
if(filters==null || filters.isEmpty())return; if(filters==null || filters.isEmpty())return;
Map<String,Field> fields=new HashMap<>(); visible(actor).forEach(f->fields.put(f.key(),f)); Map<String,Field> fields=new HashMap<>(); visible(actor).forEach(f->fields.put(f.key(),f));
for(var entry:filters.entrySet()) { for(var entry:filters.entrySet()) {
Field f=fields.get(entry.getKey()); boolean exclude=SalesOrderTypes.EXCLUDE_KEY.equals(entry.getKey());
Field f=fields.get(exclude?SalesOrderTypes.KEY:entry.getKey());
if(f==null)throw bad("筛选字段未启用或没有权限:"+entry.getKey()); if(f==null)throw bad("筛选字段未启用或没有权限:"+entry.getKey());
if(SalesOrderTypes.KEY.equals(f.key())) {
SalesOrderTypes.append(where,args,stage,entry.getValue(),SalesOrderTypes.version(jdbc),exclude);
continue;
}
boolean empty=entry.getValue().contains(EMPTY); boolean empty=entry.getValue().contains(EMPTY);
if(empty && !f.allowEmpty())throw bad(f.label()+"不允许筛选空值"); if(empty && !f.allowEmpty())throw bad(f.label()+"不允许筛选空值");
String column=stage.equals("black")?f.blackColumn():f.brightColumn(); String column=stage.equals("black")?f.blackColumn():f.brightColumn();
if(releaseId!=null&&new SourceFilterIndex(jdbc).appendLegacy(where,args,stage,releaseId,column,entry.getValue()))continue;
// Paths and selected values are bound parameters; never interpolate configured identifiers. // Paths and selected values are bound parameters; never interpolate configured identifiers.
String expression="NULLIF(NULLIF(TRIM(JSON_UNQUOTE(JSON_EXTRACT(f.raw_values, ?))), 'null'), '') COLLATE utf8mb4_0900_as_cs"; String expression="NULLIF(NULLIF(TRIM(JSON_UNQUOTE(JSON_EXTRACT(f.raw_values, ?))), 'null'), '') COLLATE utf8mb4_0900_as_cs";
List<String> values=entry.getValue().stream().filter(v->!v.equals(EMPTY)).toList(); List<String> values=entry.getValue().stream().filter(v->!v.equals(EMPTY)).toList();
@@ -69,22 +81,51 @@ public class QueryFilterFields {
} }
} }
public Map<String,Object> options(String key,String search,String releaseId,AppUserPrincipal actor) { public Map<String,Object> options(String key,String search,String releaseId,AppUserPrincipal actor) {
return options(key,search,releaseId,null,actor);
}
public Map<String,Object> options(String key,String search,String releaseId,String selectedStage,AppUserPrincipal actor) {
Field field=visible(actor).stream().filter(f->f.key().equals(key)).findFirst().orElseThrow(()->bad("筛选字段未启用或没有权限")); Field field=visible(actor).stream().filter(f->f.key().equals(key)).findFirst().orElseThrow(()->bad("筛选字段未启用或没有权限"));
if(search!=null && search.length()>256)throw bad("搜索文本过长"); if(search!=null && search.length()>256)throw bad("搜索文本过长");
if(selectedStage!=null&&!List.of("black","bright").contains(selectedStage))throw bad("阶段无效");
if(SalesOrderTypes.KEY.equals(key))return salesOrderOptions(search,releaseId,selectedStage,actor);
List<Object> args=new ArrayList<>();List<String> selects=new ArrayList<>(); List<Object> args=new ArrayList<>();List<String> selects=new ArrayList<>();
for(String stage:List.of("black","bright")) { for(String stage:List.of("black","bright")) {
String column=stage.equals("black")?field.blackColumn():field.brightColumn(); String column=stage.equals("black")?field.blackColumn():field.brightColumn();
StringBuilder scope=new StringBuilder(" WHERE rm.release_id=?"); StringBuilder scope=new StringBuilder(" WHERE rm.release_id=?");
List<Object> scoped=new ArrayList<>();scoped.add(releaseId); List<Object> scoped=new ArrayList<>();scoped.add(releaseId);
DataScope.append(scope,scoped,actor,"f.department COLLATE utf8mb4_0900_as_cs","f.group_name COLLATE utf8mb4_0900_as_cs"); DataScope.append(scope,scoped,actor,"f.department COLLATE utf8mb4_0900_as_cs","f.group_name COLLATE utf8mb4_0900_as_cs");
var index=new SourceFilterIndex(jdbc);
if(index.ready(releaseId)) {
selects.add(index.candidateSelect(stage,column,scope.toString(),scoped,args).replace("AS candidate_value","AS value"));
continue;
}
selects.add("SELECT NULLIF(NULLIF(TRIM(JSON_UNQUOTE(JSON_EXTRACT(f.raw_values, ?))), 'null'), '') COLLATE utf8mb4_0900_as_cs AS value FROM release_member rm JOIN fact_"+stage+" f ON f.build_id=rm.transform_build_id"+scope); selects.add("SELECT NULLIF(NULLIF(TRIM(JSON_UNQUOTE(JSON_EXTRACT(f.raw_values, ?))), 'null'), '') COLLATE utf8mb4_0900_as_cs AS value FROM release_member rm JOIN fact_"+stage+" f ON f.build_id=rm.transform_build_id"+scope);
args.add(path(column));args.addAll(scoped); args.add(path(column));args.addAll(scoped);
} }
String predicate=" WHERE value IS NOT NULL"; String predicate=" WHERE value IS NOT NULL AND CAST(value AS BINARY)<>CAST('null' AS BINARY)";
if(search!=null && !search.isBlank()) { predicate+=" AND LOCATE(LOWER(?),LOWER(value))>0";args.add(search); } if(search!=null && !search.isBlank()) { predicate+=" AND LOCATE(LOWER(?),LOWER(value))>0";args.add(search); }
List<String> found=jdbc.query("SELECT DISTINCT value FROM ("+String.join(" UNION ALL ",selects)+") choices"+predicate+" ORDER BY value LIMIT 201",(rs,n)->rs.getString(1),args.toArray()); List<String> found=jdbc.query("SELECT DISTINCT value FROM ("+String.join(" UNION ALL ",selects)+") choices"+predicate+" ORDER BY value LIMIT 201",(rs,n)->rs.getString(1),args.toArray());
return Map.of("values",found.stream().limit(200).toList(),"hasMore",found.size()>200); return Map.of("values",found.stream().limit(200).toList(),"hasMore",found.size()>200);
} }
private Map<String,Object> salesOrderOptions(String search,String releaseId,String selectedStage,AppUserPrincipal actor) {
String version=SalesOrderTypes.version(jdbc);
if(releaseId==null||version.isBlank())return Map.of("values",List.of(),"hasMore",false,"versionId",version);
List<Object> args=new ArrayList<>();List<String> selects=new ArrayList<>();
for(String type:SalesOrderTypes.TYPES) {
if(search!=null&&!search.isBlank()&&!type.toLowerCase(Locale.ROOT).contains(search.toLowerCase(Locale.ROOT)))continue;
args.add(type);List<String> exists=new ArrayList<>();
for(String stage:selectedStage==null?List.of("black","bright"):List.of(selectedStage)) {
StringBuilder scope=new StringBuilder(" WHERE sot.version_id=? AND sot.order_type=? AND rm.release_id=?");
List<Object> scoped=new ArrayList<>(List.of(version,type,releaseId));
DataScope.append(scope,scoped,actor,"f.department COLLATE utf8mb4_0900_as_cs","f.group_name COLLATE utf8mb4_0900_as_cs");
exists.add("EXISTS (SELECT 1 FROM sales_order_type_entry sot JOIN fact_"+stage+" f ON f.sales_order_no=sot.order_no JOIN release_member rm ON f.build_id=rm.transform_build_id AND f.source_revision_id=rm.source_revision_id"+scope+")");
args.addAll(scoped);
}
selects.add("SELECT ? AS value WHERE "+String.join(" OR ",exists));
}
List<String> values=selects.isEmpty()?List.of():jdbc.query(String.join(" UNION ALL ",selects)+" ORDER BY value",(rs,n)->rs.getString(1),args.toArray());
return Map.of("values",values,"hasMore",false,"versionId",version);
}
public static String path(String column) { return "$."+JSON.writeValueAsString(column); } public static String path(String column) { return "$."+JSON.writeValueAsString(column); }
public static ResponseStatusException bad(String text) { return new ResponseStatusException(HttpStatus.BAD_REQUEST,text); } public static ResponseStatusException bad(String text) { return new ResponseStatusException(HttpStatus.BAD_REQUEST,text); }
} }
@@ -0,0 +1,38 @@
package cn.yunsheng.quality.filters;
import cn.yunsheng.quality.governance.GovernanceAccessService;
import cn.yunsheng.quality.identity.AppUserPrincipal;
import java.util.Map;
import java.nio.charset.StandardCharsets;
import org.springframework.http.*;
import org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody;
import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.security.core.annotation.AuthenticationPrincipal;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
@RestController
@RequestMapping("/api/sales-order-types")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public class SalesOrderTypeController {
private final SalesOrderTypeService service;
private final GovernanceAccessService governance;
public SalesOrderTypeController(SalesOrderTypeService service,GovernanceAccessService governance) { this.service=service;this.governance=governance; }
private void authorize(AppUserPrincipal actor) {
governance.requireUserActivePolicy(actor);
if(!SalesOrderTypes.allowed(governance.allowedFields(actor)))throw new org.springframework.web.server.ResponseStatusException(org.springframework.http.HttpStatus.FORBIDDEN,"没有销售订单来源字段权限");
}
@GetMapping public Map<String,Object> status(@AuthenticationPrincipal AppUserPrincipal actor) { authorize(actor);return service.status(); }
@GetMapping("/export") public ResponseEntity<StreamingResponseBody> export(@AuthenticationPrincipal AppUserPrincipal actor) {
authorize(actor);
String version=service.versionForExport();
StreamingResponseBody body=output->service.writeExport(version,output);
return ResponseEntity.ok().contentType(MediaType.parseMediaType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"))
.header(HttpHeaders.CONTENT_DISPOSITION,ContentDisposition.attachment().filename("销售订单类型对应表.xlsx",StandardCharsets.UTF_8).build().toString())
.header(HttpHeaders.CACHE_CONTROL,"no-store").body(body);
}
@PostMapping("/preview") public Map<String,Object> preview(@RequestParam MultipartFile file,@RequestParam(defaultValue="MERGE") String mode,@AuthenticationPrincipal AppUserPrincipal actor) { authorize(actor);return service.preview(file,mode); }
@PostMapping("/import") public Map<String,Object> importFile(@RequestParam MultipartFile file,@RequestParam String mode,@RequestParam(defaultValue="") String expectedVersion,@RequestParam String sha256,@AuthenticationPrincipal AppUserPrincipal actor) { authorize(actor);return service.importFile(file,mode,expectedVersion,sha256,actor); }
public record Activation(String versionId,String expectedVersion) {}
@PostMapping("/activate") public Map<String,Object> activate(@RequestBody Activation input,@AuthenticationPrincipal AppUserPrincipal actor) { authorize(actor);return service.activate(input.versionId(),input.expectedVersion(),actor); }
}
@@ -0,0 +1,144 @@
package cn.yunsheng.quality.filters;
import cn.yunsheng.quality.identity.AppUserPrincipal;
import java.io.*;
import java.security.MessageDigest;
import java.util.*;
import org.apache.poi.ss.usermodel.*;
import org.apache.poi.ss.util.CellRangeAddress;
import org.apache.poi.xssf.streaming.SXSSFWorkbook;
import org.springframework.http.HttpStatus;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.multipart.MultipartFile;
import org.springframework.web.server.ResponseStatusException;
@Service
public class SalesOrderTypeService {
private final JdbcTemplate jdbc;
public SalesOrderTypeService(JdbcTemplate jdbc) { this.jdbc=jdbc; }
record Input(String filename,String sha256,LinkedHashMap<String,String> rows) {}
static Input read(MultipartFile file) {
if(file==null||file.isEmpty())throw QueryFilterFields.bad("请选择销售订单对应表");
if(file.getSize()>50L*1024*1024)throw QueryFilterFields.bad("销售订单对应表最多50MB");
try {
byte[] bytes=file.getBytes();
LinkedHashMap<String,String> rows=new LinkedHashMap<>();
try(Workbook workbook=WorkbookFactory.create(new ByteArrayInputStream(bytes))) {
if(workbook.getNumberOfSheets()!=1)throw QueryFilterFields.bad("销售订单对应表请只保留一张工作表");
Sheet sheet=workbook.getSheetAt(0); Row header=sheet.getRow(0);
if(header==null)throw QueryFilterFields.bad("对应表缺少表头");
int order=-1,type=-1;
for(Cell cell:header) {
String label=new DataFormatter().formatCellValue(cell).trim();
if(List.of("销售订单编号","销售订单号","销售订单").contains(label))order=cell.getColumnIndex();
if("销售订单类型".equals(label))type=cell.getColumnIndex();
}
if(order<0||type<0)throw QueryFilterFields.bad("对应表需要“销售订单编号”和“销售订单类型”两列");
if(sheet.getLastRowNum()>500000)throw QueryFilterFields.bad("销售订单对应表最多50万行");
for(int index=1;index<=sheet.getLastRowNum();index++) {
Row row=sheet.getRow(index);if(row==null)continue;
String key=text(row.getCell(order),index+1),value=text(row.getCell(type),index+1);
if(key.isBlank()&&value.isBlank())continue;
if(key.isBlank()||key.length()>190)throw QueryFilterFields.bad("第"+(index+1)+"行订单号为空或超过190字");
if(!SalesOrderTypes.TYPES.contains(value))throw QueryFilterFields.bad("第"+(index+1)+"行销售订单类型不属于现有7类:"+value);
String previous=rows.putIfAbsent(key,value);
if(previous!=null&&!previous.equals(value))throw QueryFilterFields.bad("第"+(index+1)+"行订单号对应多个类型:"+key);
}
}
if(rows.isEmpty())throw QueryFilterFields.bad("对应表没有有效订单");
String filename=java.nio.file.Path.of(Objects.toString(file.getOriginalFilename(),"销售订单对应表.xlsx").replace('\\','/')).getFileName().toString();
if(filename.length()>255)throw QueryFilterFields.bad("文件名过长");
return new Input(filename,HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes)),rows);
} catch(ResponseStatusException failure) { throw failure; }
catch(Exception failure) { throw QueryFilterFields.bad("无法读取对应表,请选择有效的Excel文件"); }
}
private static String text(Cell cell,int row) {
if(cell==null||cell.getCellType()==CellType.BLANK)return "";
if(cell.getCellType()!=CellType.STRING)throw QueryFilterFields.bad("第"+row+"行请使用文本保存订单号和类型,不使用数值或公式");
return cell.getStringCellValue().trim();
}
private String current(boolean lock) {
return jdbc.queryForObject("SELECT COALESCE(version_id,'') FROM sales_order_type_current WHERE dataset_id='quality'"+(lock?" FOR UPDATE":""),String.class);
}
private LinkedHashMap<String,String> entries(String version) {
LinkedHashMap<String,String> result=new LinkedHashMap<>();
if(version!=null&&!version.isBlank())jdbc.query("SELECT order_no,order_type FROM sales_order_type_entry WHERE version_id=? ORDER BY order_no",(org.springframework.jdbc.core.RowCallbackHandler)rs->result.put(rs.getString(1),rs.getString(2)),version);
return result;
}
public String versionForExport() {
String version=current(false);
if(version==null||version.isBlank())throw new ResponseStatusException(HttpStatus.NOT_FOUND,"尚未导入销售订单类型对应表");
return version;
}
@Transactional(readOnly=true)
public void writeExport(String version,OutputStream output) throws IOException {
try(SXSSFWorkbook workbook=new SXSSFWorkbook(200)) {
workbook.setCompressTempFiles(true);
Sheet sheet=workbook.createSheet("销售订单类型对应");
sheet.createFreezePane(0,1);sheet.setColumnWidth(0,28*256);sheet.setColumnWidth(1,28*256);
CellStyle textStyle=workbook.createCellStyle();textStyle.setDataFormat(workbook.createDataFormat().getFormat("@"));
sheet.setDefaultColumnStyle(0,textStyle);sheet.setDefaultColumnStyle(1,textStyle);
Row header=sheet.createRow(0);header.createCell(0).setCellValue("销售订单编号");header.createCell(1).setCellValue("销售订单类型");
int[] index={1};
// 导出开始时固定版本,后续更新对应表不会混入本次下载。订单号始终写成文本。
jdbc.query("SELECT order_no,order_type FROM sales_order_type_entry WHERE version_id=? ORDER BY order_no",(org.springframework.jdbc.core.RowCallbackHandler)rs->{
Row row=sheet.createRow(index[0]++);
for(int column=0;column<2;column++){Cell cell=row.createCell(column,CellType.STRING);cell.setCellValue(rs.getString(column+1));cell.setCellStyle(textStyle);}
},version);
sheet.setAutoFilter(new CellRangeAddress(0,index[0]-1,0,1));
workbook.write(output);
}
}
private static void requireMode(String mode) {
if(!Set.of("MERGE","REPLACE").contains(mode))throw QueryFilterFields.bad("请选择增量更新或整表替换");
}
static Map<String,Object> summary(Map<String,String> before,Map<String,String> incoming,String mode) {
requireMode(mode);int added=0,updated=0,unchanged=0;
for(var row:incoming.entrySet()) {
if(!before.containsKey(row.getKey()))added++;
else if(!before.get(row.getKey()).equals(row.getValue()))updated++;
else unchanged++;
}
long removed="REPLACE".equals(mode)?before.keySet().stream().filter(key->!incoming.containsKey(key)).count():0;
return Map.of("added",added,"updated",updated,"unchanged",unchanged,"removed",removed,"imported",incoming.size(),"total","REPLACE".equals(mode)?incoming.size():before.size()+added);
}
@Transactional(readOnly=true)
public Map<String,Object> preview(MultipartFile file,String mode) {
requireMode(mode);Input input=read(file);String base=current(false);
return Map.of("sourceFile",input.filename(),"sha256",input.sha256(),"expectedVersion",base,"summary",summary(entries(base),input.rows(),mode));
}
@Transactional
public Map<String,Object> importFile(MultipartFile file,String mode,String expected,String sha256,AppUserPrincipal actor) {
requireMode(mode);Input input=read(file);
if(!input.sha256().equals(sha256))throw QueryFilterFields.bad("文件已变化,请重新预览");
String base=current(true);
if(!Objects.equals(base,expected))throw new ResponseStatusException(HttpStatus.CONFLICT,"对应关系已更新,请重新预览");
LinkedHashMap<String,String> before=entries(base),merged=new LinkedHashMap<>();
Map<String,Object> changes=summary(before,input.rows(),mode);
if("MERGE".equals(mode))merged.putAll(before);merged.putAll(input.rows());
String id=UUID.randomUUID().toString();
jdbc.update("INSERT INTO sales_order_type_version(id,source_file,source_sha256,import_mode,row_count,created_by) VALUES(?,?,?,?,?,?)",id,input.filename(),input.sha256(),mode,merged.size(),actor.getId().toString());
jdbc.batchUpdate("INSERT INTO sales_order_type_entry(version_id,order_no,order_type) VALUES(?,?,?)",new ArrayList<>(merged.entrySet()),500,(statement,row)->{statement.setString(1,id);statement.setString(2,row.getKey());statement.setString(3,row.getValue());});
new cn.yunsheng.quality.metrics.SalesOrderMetricRollups(jdbc).prepareCurrentRelease(id);
jdbc.update("UPDATE sales_order_type_current SET version_id=? WHERE dataset_id='quality'",id);
audit(actor,"SALES_ORDER_TYPE_IMPORT",id);
return Map.of("versionId",id,"summary",changes);
}
@Transactional
public Map<String,Object> activate(String id,String expected,AppUserPrincipal actor) {
if(!Objects.equals(current(true),expected))throw new ResponseStatusException(HttpStatus.CONFLICT,"对应关系已更新,请刷新后重试");
if(jdbc.queryForObject("SELECT COUNT(*) FROM sales_order_type_version WHERE id=?",Integer.class,id)!=1)throw QueryFilterFields.bad("对应关系版本不存在");
new cn.yunsheng.quality.metrics.SalesOrderMetricRollups(jdbc).prepareCurrentRelease(id);
jdbc.update("UPDATE sales_order_type_current SET version_id=? WHERE dataset_id='quality'",id);audit(actor,"SALES_ORDER_TYPE_RESTORE",id);
return Map.of("versionId",id);
}
public Map<String,Object> status() {
return Map.of("versionId",current(false),"versions",jdbc.queryForList("SELECT id,source_file AS sourceFile,row_count AS rowCount,import_mode AS importMode,created_at AS createdAt FROM sales_order_type_version ORDER BY created_at DESC LIMIT 30"),"types",SalesOrderTypes.TYPES);
}
private void audit(AppUserPrincipal actor,String action,String version) {
jdbc.update("INSERT INTO audit_event(actor_user_id,action_code,target_type,target_id,outcome,details) VALUES(?,?,'sales_order_type_version',?,'SUCCESS',JSON_OBJECT('versionId',?))",actor.getId().toString(),action,version,version);
}
}
@@ -0,0 +1,67 @@
package cn.yunsheng.quality.filters;
import java.util.*;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.web.context.request.RequestContextHolder;
import org.springframework.web.context.request.RequestAttributes;
/** A reference snapshot, independent of quality-data releases. Never changes raw source values. */
public final class SalesOrderTypes {
public static final String KEY = "salesOrderType";
public static final String EXCLUDE_KEY = "salesOrderTypeNotIn";
public static final String LABEL = "销售订单类型";
public static final List<String> TYPES = List.of("标准销售订单", "样品销售订单", "国内寄售补货订单", "备料销售订单", "内部研发销售订单", "免费补货订单", "外销寄售补货订单");
private static final String PIN = SalesOrderTypes.class.getName() + ".version";
private SalesOrderTypes() {}
public static boolean allowed(Set<String> fields) {
return fields.contains("*") || fields.containsAll(List.of("销售订单号", "销售订单"));
}
public static String column(String stage) {
return switch(stage) { case "black" -> "销售订单号"; case "bright" -> "销售订单"; default -> throw QueryFilterFields.bad("阶段无效"); };
}
public static String version(JdbcTemplate jdbc) {
RequestAttributes request = RequestContextHolder.getRequestAttributes();
if (request != null && request.getAttribute(PIN, RequestAttributes.SCOPE_REQUEST) instanceof String pinned) return pinned;
String version = jdbc.queryForObject("SELECT COALESCE(version_id,'') FROM sales_order_type_current WHERE dataset_id='quality'", String.class);
version = version == null ? "" : version;
if (request != null) request.setAttribute(PIN, version, RequestAttributes.SCOPE_REQUEST);
return version;
}
public static String expression(String stage, String version, List<Object> args) {
column(stage);args.add(version);
return "(SELECT sot.order_type FROM sales_order_type_entry sot WHERE sot.version_id=? AND sot.order_no="
+ "f.sales_order_no)";
}
public static void append(StringBuilder where,List<Object> args,String stage,List<String> selected,String version) {
append(where,args,stage,selected,version,false);
}
public static void append(StringBuilder where,List<Object> args,String stage,List<String> selected,String version,boolean exclude) {
List<String> types=selected.stream().filter(value->!QueryFilterFields.EMPTY.equals(value)).toList();
if(types.stream().anyMatch(value->!TYPES.contains(value)))throw QueryFilterFields.bad("销售订单类型无效");
if(selected.isEmpty())return;
if(exclude) {
column(stage);where.append(" AND (");
if(!types.isEmpty()) {
args.add(version);where.append("NOT EXISTS (SELECT 1 FROM sales_order_type_entry sot WHERE sot.version_id=? AND sot.order_no=f.sales_order_no AND sot.order_type IN (").append(String.join(",",Collections.nCopies(types.size(),"?"))).append("))");
args.addAll(types);
}
if(selected.contains(QueryFilterFields.EMPTY)) {
if(!types.isEmpty())where.append(" AND ");
args.add(version);where.append("EXISTS (SELECT 1 FROM sales_order_type_entry sot WHERE sot.version_id=? AND sot.order_no=f.sales_order_no)");
}
where.append(')');return;
}
where.append(" AND (");
if(!types.isEmpty()) {
column(stage);args.add(version);
where.append("EXISTS (SELECT 1 FROM sales_order_type_entry sot WHERE sot.version_id=? AND sot.order_no=f.sales_order_no AND sot.order_type IN (").append(String.join(",",Collections.nCopies(types.size(),"?"))).append("))");
args.addAll(types);
}
if(selected.contains(QueryFilterFields.EMPTY)) {
if(!types.isEmpty())where.append(" OR ");
args.add(version);where.append("NOT EXISTS (SELECT 1 FROM sales_order_type_entry sot WHERE sot.version_id=? AND sot.order_no=f.sales_order_no)");
}
where.append(')');
}
}
@@ -60,8 +60,18 @@ public class SourceColumnQueries {
} }
return List.copyOf(result); return List.copyOf(result);
} }
// 只提取一次,并区分 JSON null 与字符串“null”,供筛选、候选值和成员读取共用。
private static final String SOURCE_TEXT = "JSON_UNQUOTE(NULLIF(JSON_EXTRACT(f.raw_values, ?), CAST('null' AS JSON)))";
private static final String SOURCE_VALUE = "NULLIF(TRIM("+SOURCE_TEXT+"), '') COLLATE utf8mb4_0900_as_cs";
/** 原始字段筛选按发布构建读取,避免优化器从整年月份范围扫描大 JSON 原表。 */
public static String factIndexHint(String stage) {
if(!List.of("black","bright").contains(stage))throw bad("数据阶段无效");
return " FORCE INDEX (uq_fact_"+stage+"_origin)";
}
private static final List<String> STAGES = List.of("black", "bright"); private static final List<String> STAGES = List.of("black", "bright");
private static final List<String> OPERATORS = List.of("in", "not_in", "is_empty", "not_empty"); private static final List<String> OPERATORS = List.of("in", "contains", "not_in", "is_empty", "not_empty");
private static final Map<String, String> CANONICAL_COLUMNS = Map.of( private static final Map<String, String> CANONICAL_COLUMNS = Map.of(
"product", "product", "department", "department", "group", "group_name", "projectGroup", "project_group"); "product", "product", "department", "department", "group", "group_name", "projectGroup", "project_group");
private static final Map<String, String> CANONICAL_LABELS = Map.of( private static final Map<String, String> CANONICAL_LABELS = Map.of(
@@ -70,11 +80,17 @@ public class SourceColumnQueries {
private final JdbcTemplate jdbc; private final JdbcTemplate jdbc;
private final GovernanceAccessService governance; private final GovernanceAccessService governance;
private final JsonMapper json; private final JsonMapper json;
private SourceFilterIndex filterIndex;
@org.springframework.beans.factory.annotation.Autowired
public void setFilterIndex(SourceFilterIndex index) { this.filterIndex=index; }
public boolean indexed(String release) { return filterIndex.ready(release); }
public SourceColumnQueries(JdbcTemplate jdbc, GovernanceAccessService governance, JsonMapper json) { public SourceColumnQueries(JdbcTemplate jdbc, GovernanceAccessService governance, JsonMapper json) {
this.jdbc = jdbc; this.jdbc = jdbc;
this.governance = governance; this.governance = governance;
this.json = json; this.json = json;
this.filterIndex = new SourceFilterIndex(jdbc);
} }
/** /**
@@ -94,16 +110,16 @@ public class SourceColumnQueries {
Map<String, Boolean> stagePresent = new LinkedHashMap<>(); Map<String, Boolean> stagePresent = new LinkedHashMap<>();
for (String stage : STAGES) { for (String stage : STAGES) {
Set<String> headers = new LinkedHashSet<>(); Set<String> headers = new LinkedHashSet<>();
StringBuilder representativeScope = new StringBuilder(" WHERE rm.release_id=?"); StringBuilder representativeScope = new StringBuilder(" WHERE f2.build_id=rm.transform_build_id AND f2.source_revision_id=rm.source_revision_id");
List<Object> args = new ArrayList<>(List.of(releaseId)); List<Object> args = new ArrayList<>(List.of(releaseId));
DataScope.append(representativeScope, args, actor, DataScope.append(representativeScope, args, actor,
"f2.department COLLATE utf8mb4_0900_as_cs", "f2.group_name COLLATE utf8mb4_0900_as_cs"); "f2.department COLLATE utf8mb4_0900_as_cs", "f2.group_name COLLATE utf8mb4_0900_as_cs");
// Stop at the first authorized fact of each build instead of aggregating every fact id.
String sql = "SELECT CAST(JSON_KEYS(f.raw_values) AS CHAR) " String sql = "SELECT CAST(JSON_KEYS(f.raw_values) AS CHAR) "
+ "FROM (SELECT f2.build_id,f2.source_revision_id,MIN(f2.id) AS representative_id " + "FROM (SELECT DISTINCT transform_build_id,source_revision_id FROM release_member WHERE release_id=?) rm "
+ "FROM release_member rm JOIN fact_" + stage + " f2 " + "JOIN fact_" + stage + " f ON f.id=(SELECT f2.id FROM fact_" + stage + " f2"
+ "ON f2.build_id=rm.transform_build_id AND f2.source_revision_id=rm.source_revision_id" + (filterIndex.available()?" FORCE INDEX (ix_"+stage+"_source_catalog)":factIndexHint(stage))
+ representativeScope + " GROUP BY f2.build_id,f2.source_revision_id) representative " + representativeScope + " ORDER BY f2.build_id,f2.source_revision_id,f2.id LIMIT 1) ORDER BY rm.transform_build_id";
+ "JOIN fact_" + stage + " f ON f.id=representative.representative_id ORDER BY representative.build_id";
List<String> keyLists = rawHeaders.get(new QueryKey(sql,args,"raw-headers"), List<String> keyLists = rawHeaders.get(new QueryKey(sql,args,"raw-headers"),
() -> Collections.unmodifiableList(new ArrayList<>(jdbc.query(sql, (rs, row) -> rs.getString(1), args.toArray())))); () -> Collections.unmodifiableList(new ArrayList<>(jdbc.query(sql, (rs, row) -> rs.getString(1), args.toArray()))));
stagePresent.put(stage, !keyLists.isEmpty()); stagePresent.put(stage, !keyLists.isEmpty());
@@ -192,12 +208,13 @@ public class SourceColumnQueries {
normalizedStages.add(value); normalizedStages.add(value);
} }
List<String> values = new ArrayList<>(); List<String> values = new ArrayList<>();
if (operator.equals("in") || operator.equals("not_in")) { if (operator.equals("in") || operator.equals("not_in") || operator.equals("contains")) {
if (!(rule.get("values") instanceof List<?> rawValues) || rawValues.isEmpty() || rawValues.size() > 50) if (!(rule.get("values") instanceof List<?> rawValues) || rawValues.isEmpty() || rawValues.size() > 50)
throw bad("in/not_in 筛选必须提供1到50个字符串值"); throw bad("文本筛选必须提供1到50个字符串值");
for (Object value : rawValues) { for (Object value : rawValues) {
if (!(value instanceof String text) || text.isEmpty() || text.length() > 256) if (!(value instanceof String text) || text.isEmpty() || text.length() > 256)
throw bad("来源列筛选值必须是1到256字符的完整字符串;空值请使用空值操作符"); throw bad("来源列筛选值必须是1到256字符的完整字符串;空值请使用空值操作符");
if(operator.equals("contains")&&text.isBlank())throw bad("包含筛选的关键词不能全为空白");
if (!values.contains(text)) values.add(text); if (!values.contains(text)) values.add(text);
} }
} else if (rule.containsKey("values")) { } else if (rule.containsKey("values")) {
@@ -207,7 +224,7 @@ public class SourceColumnQueries {
normalized.put("field", field); normalized.put("field", field);
normalized.put("stages", List.copyOf(normalizedStages)); normalized.put("stages", List.copyOf(normalizedStages));
normalized.put("operator", operator); normalized.put("operator", operator);
if (operator.equals("in") || operator.equals("not_in")) normalized.put("values", List.copyOf(values)); if (operator.equals("in") || operator.equals("not_in") || operator.equals("contains")) normalized.put("values", List.copyOf(values));
parsed.add(Collections.unmodifiableMap(normalized)); parsed.add(Collections.unmodifiableMap(normalized));
} }
return List.copyOf(parsed); return List.copyOf(parsed);
@@ -239,11 +256,10 @@ public class SourceColumnQueries {
String operator = (String) rule.get("operator"); String operator = (String) rule.get("operator");
String expression; String expression;
Object column = ((Map<?, ?>) field.get("columns")).get(stage); Object column = ((Map<?, ?>) field.get("columns")).get(stage);
@SuppressWarnings("unchecked") List<String> selected = (List<String>) rule.getOrDefault("values",List.of());
if("source_value".equals(field.get("type")) && filterIndex.append(where,args,stage,releaseId,String.valueOf(column),operator,selected))continue;
if ("source_value".equals(field.get("type"))) { if ("source_value".equals(field.get("type"))) {
expression = "CASE WHEN JSON_EXTRACT(f.raw_values, ?) IS NULL " expression = SOURCE_VALUE;
+ "OR JSON_TYPE(JSON_EXTRACT(f.raw_values, ?))='NULL' THEN NULL "
+ "ELSE NULLIF(TRIM(JSON_UNQUOTE(JSON_EXTRACT(f.raw_values, ?))), '') END "
+ "COLLATE utf8mb4_0900_as_cs";
} else { } else {
String factColumn = CANONICAL_COLUMNS.get(key); String factColumn = CANONICAL_COLUMNS.get(key);
if (factColumn == null) throw bad("规范维度无效"); if (factColumn == null) throw bad("规范维度无效");
@@ -255,19 +271,26 @@ public class SourceColumnQueries {
if (operator.equals("in")) { if (operator.equals("in")) {
where.append(expression).append(" IN (") where.append(expression).append(" IN (")
.append(String.join(",", Collections.nCopies(values.size(), "?"))).append(')'); .append(String.join(",", Collections.nCopies(values.size(), "?"))).append(')');
if ("source_value".equals(field.get("type"))) bindSourcePaths(args, String.valueOf(column)); if ("source_value".equals(field.get("type"))) bindSourcePath(args, String.valueOf(column));
} else { } else {
where.append('(').append(expression).append(" IS NULL OR ").append(expression).append(" NOT IN (") where.append('(').append(expression).append(" IS NULL OR ").append(expression).append(" NOT IN (")
.append(String.join(",", Collections.nCopies(values.size(), "?"))).append("))"); .append(String.join(",", Collections.nCopies(values.size(), "?"))).append("))");
if ("source_value".equals(field.get("type"))) { if ("source_value".equals(field.get("type"))) {
bindSourcePaths(args, String.valueOf(column)); bindSourcePath(args, String.valueOf(column));
bindSourcePaths(args, String.valueOf(column)); bindSourcePath(args, String.valueOf(column));
} }
} }
args.addAll(values); args.addAll(values);
} else if (operator.equals("contains")) {
for(int i=0;i<selected.size();i++) {
if(i>0)where.append(" OR ");
where.append("LOCATE(?,").append(expression).append(")>0");
args.add(selected.get(i));
if("source_value".equals(field.get("type")))bindSourcePath(args,String.valueOf(column));
}
} else { } else {
where.append(expression).append(operator.equals("is_empty") ? " IS NULL" : " IS NOT NULL"); where.append(expression).append(operator.equals("is_empty") ? " IS NULL" : " IS NOT NULL");
if ("source_value".equals(field.get("type"))) bindSourcePaths(args, String.valueOf(column)); if ("source_value".equals(field.get("type"))) bindSourcePath(args, String.valueOf(column));
} }
where.append(')'); where.append(')');
} }
@@ -376,12 +399,16 @@ public class SourceColumnQueries {
public Map<String, Object> values(String releaseId, Map<String, Object> context, String fieldKey, public Map<String, Object> values(String releaseId, Map<String, Object> context, String fieldKey,
List<String> stages, String search, int limit, int offset, List<String> stages, String search, int limit, int offset,
AppUserPrincipal actor) { AppUserPrincipal actor) {
return values(releaseId,context,fieldKey,stages,search,limit,offset,false,actor);
}
public Map<String,Object> values(String releaseId,Map<String,Object> context,String fieldKey,List<String> stages,
String search,int limit,int offset,boolean candidateOnly,AppUserPrincipal actor) {
requireQueryAccess(actor); requireQueryAccess(actor);
return snapshot(()->valuesSnapshot(releaseId,context,fieldKey,stages,search,limit,offset,actor)); return snapshot(()->valuesSnapshot(releaseId,context,fieldKey,stages,search,limit,offset,candidateOnly,actor));
} }
private Map<String,Object> valuesSnapshot(String releaseId,Map<String,Object> context,String fieldKey, private Map<String,Object> valuesSnapshot(String releaseId,Map<String,Object> context,String fieldKey,
List<String> stages,String search,int limit,int offset,AppUserPrincipal actor) { List<String> stages,String search,int limit,int offset,boolean candidateOnly,AppUserPrincipal actor) {
if (releaseId == null || releaseId.isBlank()) throw bad("正式版标识无效"); if (releaseId == null || releaseId.isBlank()) throw bad("正式版标识无效");
if (fieldKey == null || fieldKey.isBlank()) throw bad("来源列标识无效"); if (fieldKey == null || fieldKey.isBlank()) throw bad("来源列标识无效");
if (stages == null || stages.isEmpty() || stages.size() > STAGES.size() if (stages == null || stages.isEmpty() || stages.size() > STAGES.size()
@@ -412,6 +439,10 @@ public class SourceColumnQueries {
StringBuilder where = new StringBuilder(" WHERE rm.release_id=?"); StringBuilder where = new StringBuilder(" WHERE rm.release_id=?");
List<Object> whereArgs = new ArrayList<>(List.of(releaseId)); List<Object> whereArgs = new ArrayList<>(List.of(releaseId));
appendContextFilters(where, whereArgs, stage, releaseId, queryContext, actor); appendContextFilters(where, whereArgs, stage, releaseId, queryContext, actor);
if("source_value".equals(field.get("type")) && filterIndex.ready(releaseId)) {
selects.add(filterIndex.candidateSelect(stage,String.valueOf(columns.get(stage)),where.toString(),whereArgs,unionArgs));
continue;
}
String fact = stage.equals("black") ? "fact_black" : "fact_bright"; String fact = stage.equals("black") ? "fact_black" : "fact_bright";
selects.add("SELECT " + expression + " AS candidate_value " selects.add("SELECT " + expression + " AS candidate_value "
+ "FROM release_member rm JOIN source_revision sr ON sr.id=rm.source_revision_id " + "FROM release_member rm JOIN source_revision sr ON sr.id=rm.source_revision_id "
@@ -430,6 +461,13 @@ public class SourceColumnQueries {
} }
String candidates = "SELECT candidate_value FROM (" + union + ") source_values" + condition; String candidates = "SELECT candidate_value FROM (" + union + ") source_values" + condition;
String grouped = "SELECT candidate_value FROM (" + candidates + ") matches GROUP BY candidate_value"; String grouped = "SELECT candidate_value FROM (" + candidates + ") matches GROUP BY candidate_value";
if(candidateOnly) {
String sql="SELECT candidate_value FROM ("+candidates+") matches GROUP BY candidate_value ORDER BY candidate_value LIMIT ? OFFSET ?";
List<Object> parameters=combine(unionArgs,searchArgs);parameters.add(limit+1);parameters.add(offset);
List<Map<String,Object>> found=pages.get(new QueryKey(sql,parameters,"candidates:"+fieldKey),()->List.copyOf(jdbc.query(sql,(rs,row)->Map.of("value",rs.getString(1)),parameters.toArray())));
boolean more=found.size()>limit;
return Map.of("rows",List.copyOf(found.subList(0,Math.min(limit,found.size()))),"hasMore",more,"limit",limit,"offset",offset,"pageComplete",true,"complete",offset==0&&!more,"releaseId",releaseId,"field",fieldKey,"selectedStages",List.copyOf(stages));
}
long total = total("SELECT COUNT(*) FROM (" + grouped + ") distinct_values", combine(unionArgs,searchArgs)); long total = total("SELECT COUNT(*) FROM (" + grouped + ") distinct_values", combine(unionArgs,searchArgs));
List<Map<String, Object>> rows = page("SELECT candidate_value FROM (" + candidates + ") matches " List<Map<String, Object>> rows = page("SELECT candidate_value FROM (" + candidates + ") matches "
+ "GROUP BY candidate_value ORDER BY candidate_value LIMIT ? OFFSET ?", + "GROUP BY candidate_value ORDER BY candidate_value LIMIT ? OFFSET ?",
@@ -556,7 +594,7 @@ public class SourceColumnQueries {
List<String> groups = parseSelections(context.dimensions().get("group"), "group"); List<String> groups = parseSelections(context.dimensions().get("group"), "group");
if (!context.includeUnassigned() && !groups.contains("__UNASSIGNED__")) if (!context.includeUnassigned() && !groups.contains("__UNASSIGNED__"))
where.append(" AND (f.group_name IS NOT NULL AND f.group_name<>'')"); where.append(" AND (f.group_name IS NOT NULL AND f.group_name<>'')");
new QueryFilterFields(jdbc, governance).append(where, args, stage, context.extraFilters(), actor); new QueryFilterFields(jdbc, governance).append(where, args, stage, context.extraFilters(), actor,releaseId);
if (!context.columnFilters().isEmpty()) append(where, args, stage, context.columnFilters(), releaseId, actor); if (!context.columnFilters().isEmpty()) append(where, args, stage, context.columnFilters(), releaseId, actor);
DataScope.append(where, args, actor, DataScope.append(where, args, actor,
"f.department COLLATE utf8mb4_0900_as_cs", "f.group_name COLLATE utf8mb4_0900_as_cs"); "f.department COLLATE utf8mb4_0900_as_cs", "f.group_name COLLATE utf8mb4_0900_as_cs");
@@ -578,12 +616,7 @@ public class SourceColumnQueries {
} }
String path = jsonPath(column); String path = jsonPath(column);
args.add(path); args.add(path);
args.add(path); return SOURCE_VALUE;
args.add(path);
return "CASE WHEN JSON_EXTRACT(f.raw_values, ?) IS NULL "
+ "OR JSON_TYPE(JSON_EXTRACT(f.raw_values, ?))='NULL' THEN NULL "
+ "ELSE NULLIF(TRIM(JSON_UNQUOTE(JSON_EXTRACT(f.raw_values, ?))), '') END "
+ "COLLATE utf8mb4_0900_as_cs";
} }
private static List<Object> combine(List<Object> first, List<Object> second) { private static List<Object> combine(List<Object> first, List<Object> second) {
@@ -601,9 +634,7 @@ public class SourceColumnQueries {
} }
String path = jsonPath(column); String path = jsonPath(column);
args.add(path); args.add(path);
args.add(path); return SOURCE_TEXT+" COLLATE utf8mb4_0900_as_cs";
return "CASE WHEN JSON_TYPE(JSON_EXTRACT(f.raw_values, ?))='NULL' THEN NULL "
+ "ELSE JSON_UNQUOTE(JSON_EXTRACT(f.raw_values, ?)) END COLLATE utf8mb4_0900_as_cs";
} }
private static String tupleColumns(int count) { private static String tupleColumns(int count) {
@@ -814,11 +845,9 @@ public class SourceColumnQueries {
catch (Exception e) { throw new IllegalArgumentException("来源列名无法编码", e); } catch (Exception e) { throw new IllegalArgumentException("来源列名无法编码", e); }
} }
private void bindSourcePaths(List<Object> args, String header) { private void bindSourcePath(List<Object> args, String header) {
String path = jsonPath(header); String path = jsonPath(header);
args.add(path); args.add(path);
args.add(path);
args.add(path);
} }
private static ResponseStatusException bad(String message) { private static ResponseStatusException bad(String message) {
@@ -0,0 +1,143 @@
package cn.yunsheng.quality.filters;
import java.util.List;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.support.TransactionTemplate;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
/** Sparse, build-owned source values. Every consumer still authorizes and scopes the actual facts. */
@Service
public class SourceFilterIndex {
public record BuildAvailable(String buildId) {}
private static final int FORMAT = 1;
static final String CREATE_STAGE_SQL="CREATE TEMPORARY TABLE source_filter_staging(field_id BIGINT NOT NULL,fact_id BIGINT NOT NULL,value LONGTEXT COLLATE utf8mb4_0900_as_cs NOT NULL,value_hash BINARY(32) NOT NULL,KEY(field_id,value_hash,fact_id)) ENGINE=InnoDB";
static final String MEMBER_INSERT_SQL="INSERT INTO source_filter_member(field_id,fact_id,value_id) SELECT s.field_id,s.fact_id,v.id FROM source_filter_staging s STRAIGHT_JOIN source_filter_value v ON v.field_id=s.field_id AND v.value_hash=s.value_hash";
static final String COLLISION_SQL="SELECT COUNT(*) FROM source_filter_staging s STRAIGHT_JOIN source_filter_value v ON v.field_id=s.field_id AND v.value_hash=s.value_hash WHERE CAST(v.value AS BINARY)<>CAST(s.value AS BINARY)";
private final JdbcTemplate jdbc;
private volatile Boolean available;
private volatile Boolean candidateScopeAvailable;
public SourceFilterIndex(JdbcTemplate jdbc) { this.jdbc = jdbc; }
public boolean available() {
if (available == null) {
Integer count=jdbc.queryForObject("SELECT COUNT(*) FROM information_schema.tables WHERE table_schema=DATABASE() AND table_name IN ('source_filter_build','source_filter_field','source_filter_value','source_filter_member')",Integer.class);
available=count!=null&&count==4;
}
return available;
}
private String candidateFactHint(String stage) {
if (candidateScopeAvailable == null) {
Integer count=jdbc.queryForObject("SELECT COUNT(DISTINCT table_name) FROM information_schema.statistics WHERE table_schema=DATABASE() AND ((table_name='fact_black' AND index_name='ix_black_candidate_scope') OR (table_name='fact_bright' AND index_name='ix_bright_candidate_scope'))",Integer.class);
candidateScopeAvailable=count!=null&&count==2;
}
return candidateScopeAvailable ? " FORCE INDEX (ix_"+stage+"_candidate_scope)" : "";
}
public boolean ready(String release) {
if(release==null||!available())return false;
// Only successful immutable builds are retained; a pending backfill is checked again later.
Integer missing=jdbc.queryForObject("SELECT COUNT(*) FROM release_member rm LEFT JOIN source_filter_build sb ON sb.build_id=rm.transform_build_id AND sb.format_version=? WHERE rm.release_id=? AND sb.build_id IS NULL",Integer.class,FORMAT,release);
if(missing==null||missing!=0)return false;
return true;
}
/** Idempotent and atomic with fact construction, including callers which instantiate this helper. */
public void ensureBuild(String build) {
if(!available())return;
var tx=new TransactionTemplate(new DataSourceTransactionManager(jdbc.getDataSource()));
tx.executeWithoutResult(status->{
jdbc.queryForObject("SELECT id FROM transform_build WHERE id=? FOR UPDATE",String.class,build);
Integer done=jdbc.queryForObject("SELECT COUNT(*) FROM source_filter_build WHERE build_id=? AND format_version=?",Integer.class,build,FORMAT);
if(done!=null&&done>0)return;
jdbc.update("DELETE m FROM source_filter_member m JOIN source_filter_field sf ON sf.id=m.field_id WHERE sf.build_id=?",build);
jdbc.update("DELETE v FROM source_filter_value v JOIN source_filter_field sf ON sf.id=v.field_id WHERE sf.build_id=?",build);
jdbc.update("DELETE FROM source_filter_field WHERE build_id=?",build);
jdbc.update("DELETE FROM source_filter_build WHERE build_id=?",build);
// Only staging is bounded to 500 facts. The build transaction and persisted dictionary scale with the source.
try {
for(String stage:List.of("black","bright"))buildStage(build,stage);
jdbc.update("INSERT INTO source_filter_build(build_id,format_version) VALUES(?,?)",build,FORMAT);
} finally { jdbc.execute("DROP TEMPORARY TABLE IF EXISTS source_filter_staging"); }
});
}
private void buildStage(String build,String stage) {
var bounds=jdbc.queryForMap("SELECT MIN(id) AS first_id,MAX(id) AS last_id FROM fact_"+stage+" FORCE INDEX(uq_fact_"+stage+"_origin) WHERE build_id=?",build);
if(bounds.get("first_id")==null)return;
long after=((Number)bounds.get("first_id")).longValue()-1;
long last=((Number)bounds.get("last_id")).longValue();
while(after<last) {
if(Thread.currentThread().isInterrupted())throw new IllegalStateException("来源筛选索引构建已中断");
List<Long> ids=jdbc.queryForList("SELECT id FROM fact_"+stage+" FORCE INDEX(PRIMARY) WHERE build_id=? AND id>? AND id<=? ORDER BY id LIMIT 500",Long.class,build,after,last);
if(ids.isEmpty())return;
long end=ids.getLast();
String rows=" FROM fact_"+stage+" f FORCE INDEX(PRIMARY) JOIN JSON_TABLE(JSON_KEYS(f.raw_values),'$[*]' COLUMNS(column_name LONGTEXT PATH '$' ERROR ON ERROR)) k ON TRUE WHERE f.build_id=? AND f.id>? AND f.id<=?";
jdbc.update("INSERT IGNORE INTO source_filter_field(build_id,stage,column_name,name_hash) SELECT DISTINCT ?,?,k.column_name,UNHEX(SHA2(CAST(k.column_name AS BINARY),256))"+rows,build,stage,build,after,end);
String expression="NULLIF(TRIM(JSON_UNQUOTE(NULLIF(JSON_EXTRACT(f.raw_values,CONCAT('$.',JSON_QUOTE(k.column_name))),CAST('null' AS JSON)))),'')";
String extracted="SELECT sf.id AS field_id,f.id AS fact_id,"+expression+" AS value FROM fact_"+stage+" f FORCE INDEX(PRIMARY) JOIN JSON_TABLE(JSON_KEYS(f.raw_values),'$[*]' COLUMNS(column_name LONGTEXT PATH '$' ERROR ON ERROR)) k ON TRUE JOIN source_filter_field sf ON sf.build_id=f.build_id AND sf.stage=? AND sf.name_hash=UNHEX(SHA2(CAST(k.column_name AS BINARY),256)) AND sf.column_name=k.column_name COLLATE utf8mb4_0900_bin WHERE f.build_id=? AND f.id>? AND f.id<=?";
// DROP/CREATE TEMPORARY do not commit the owning build transaction. Reset the
// physical table too: DELETE retains undo/deleted records until that transaction ends.
jdbc.execute("DROP TEMPORARY TABLE IF EXISTS source_filter_staging");
jdbc.execute(CREATE_STAGE_SQL);
jdbc.update("INSERT INTO source_filter_staging SELECT /*+ NO_MERGE(extracted) */ field_id,fact_id,value,UNHEX(SHA2(CAST(value AS BINARY),256)) FROM ("+extracted+") extracted WHERE value IS NOT NULL",stage,build,after,end);
jdbc.update("INSERT INTO source_filter_value(field_id,value_hash,value) SELECT s.field_id,s.value_hash,ANY_VALUE(s.value) FROM source_filter_staging s LEFT JOIN source_filter_value v ON v.field_id=s.field_id AND v.value_hash=s.value_hash WHERE v.id IS NULL GROUP BY s.field_id,s.value_hash");
Integer collisions=jdbc.queryForObject(COLLISION_SQL,Integer.class);
if(collisions!=null&&collisions>0)throw new IllegalStateException("来源筛选字典校验失败,索引未发布");
jdbc.update(MEMBER_INSERT_SQL);
after=end;
}
}
/** Same-row membership is shared by metrics, trends, details and exports; empty values are sparse. */
public boolean append(StringBuilder where,List<Object> args,String stage,String release,String column,
String operator,List<String> values) {
if(!ready(release))return false;
where.append(" AND ");
membership(where,args,stage,release,column,operator.equals("not_in")||operator.equals("is_empty"),
operator.equals("in")||operator.equals("not_in")||operator.equals("contains")?values:null,false,operator.equals("contains"));
return true;
}
/** Legacy configured filters classify only the byte-exact literal string "null" as empty. */
public boolean appendLegacy(StringBuilder where,List<Object> args,String stage,String release,String column,List<String> selected) {
if(!ready(release))return false;
var values=selected.stream().filter(v->!v.equals(QueryFilterFields.EMPTY)).toList();
where.append(" AND (");
if(values.isEmpty())where.append("FALSE");
else membership(where,args,stage,release,column,false,values,true,false);
if(selected.contains(QueryFilterFields.EMPTY)) {
where.append(" OR ");membership(where,args,stage,release,column,true,null,true,false);
}
where.append(')');return true;
}
private void membership(StringBuilder where,List<Object> args,String stage,String release,String column,
boolean negative,List<String> values,boolean legacy,boolean contains) {
where.append("f.id ").append(negative?"NOT IN (":"IN (")
.append("SELECT sm.fact_id FROM source_filter_field sf JOIN release_member srm ON srm.transform_build_id=sf.build_id JOIN source_filter_value sv ON sv.field_id=sf.id JOIN source_filter_member sm ON sm.value_id=sv.id WHERE srm.release_id=? AND sf.stage=? AND sf.column_name=? COLLATE utf8mb4_0900_bin");
args.add(release);args.add(stage);args.add(column);
if(legacy)where.append(" AND CAST(sv.value AS BINARY)<>CAST('null' AS BINARY)");
if(contains) {
where.append(" AND (");
for(int i=0;i<values.size();i++) {
if(i>0)where.append(" OR ");
where.append("LOCATE(?,sv.value COLLATE utf8mb4_0900_as_cs)>0");args.add(values.get(i));
}
where.append(')');
} else if(values!=null) {
where.append(" AND sv.value COLLATE utf8mb4_0900_as_cs IN (").append(String.join(",",java.util.Collections.nCopies(values.size(),"?"))).append(')');
args.addAll(values);
}
where.append(')');
}
/** Dictionary-first candidates short-circuit after one authorized matching member per value. */
public String candidateSelect(String stage,String column,String scope,List<Object> scopeArgs,List<Object> args) {
args.add(scopeArgs.getFirst());args.add(column);args.add(stage);args.addAll(scopeArgs);
return "SELECT sv.value AS candidate_value FROM release_member frm JOIN source_filter_field sf ON sf.build_id=frm.transform_build_id JOIN source_filter_value sv ON sv.field_id=sf.id "
+"WHERE frm.release_id=? AND sf.column_name=? COLLATE utf8mb4_0900_bin AND sf.stage=? AND EXISTS (SELECT 1 FROM source_filter_member sm JOIN fact_"+stage+" f"+candidateFactHint(stage)+" ON f.id=sm.fact_id JOIN release_member rm ON rm.transform_build_id=f.build_id AND rm.source_revision_id=f.source_revision_id JOIN source_revision sr ON sr.id=rm.source_revision_id "
+scope+" AND sm.value_id=sv.id AND f.build_id=sf.build_id)";
}
}
@@ -0,0 +1,194 @@
package cn.yunsheng.quality.filters;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.RejectedExecutionException;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import jakarta.annotation.PreDestroy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.context.event.EventListener;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Component;
/** Asynchronously builds current and newly completed snapshots; unindexed queries retain their raw-value fallback. */
@Component
public class SourceFilterIndexBackfill implements ApplicationRunner {
private static final Logger LOG = LoggerFactory.getLogger(SourceFilterIndexBackfill.class);
private static final int MAX_PENDING_BUILDS = 1024;
private final JdbcTemplate jdbc;
private final SourceFilterIndex index;
private final ExecutorService worker;
private final Object pendingLock = new Object();
private final LinkedHashSet<String> pending = new LinkedHashSet<>();
private boolean drainScheduled;
private String inFlight;
@Autowired
public SourceFilterIndexBackfill(JdbcTemplate jdbc, SourceFilterIndex index) {
this(jdbc, index, createWorker());
}
SourceFilterIndexBackfill(JdbcTemplate jdbc, SourceFilterIndex index, ExecutorService worker) {
this.jdbc = jdbc;
this.index = index;
this.worker = worker;
}
@Override
public void run(ApplicationArguments args) {
List<String> currentBuilds;
try {
if (!index.available()) return;
currentBuilds = jdbc.query("""
SELECT DISTINCT b.id
FROM current_release c
JOIN data_release r ON r.id=c.release_id
JOIN release_member rm ON rm.release_id=r.id
JOIN transform_build b ON b.id=rm.transform_build_id
WHERE c.dataset_id='quality'
AND r.status='PUBLISHED'
AND r.complete=TRUE
AND b.status='COMPLETED'
ORDER BY b.id
""", (rs, row) -> rs.getString(1));
} catch (RuntimeException failure) {
LOG.error("Source filter index backfill discovery failed; raw-value fallback remains available", failure);
return;
}
if (currentBuilds.isEmpty()) return;
int accepted = enqueue(currentBuilds);
LOG.info("Source filter index backfill queued: discovered={}, accepted={}", currentBuilds.size(), accepted);
}
@EventListener
public void onBuildAvailable(SourceFilterIndex.BuildAvailable event) {
if (event == null || event.buildId() == null || event.buildId().isBlank()) return;
int accepted = enqueue(List.of(event.buildId()));
if (accepted > 0) LOG.info("Source filter index build queued after commit: build={}", event.buildId());
}
private int enqueue(List<String> builds) {
boolean schedule = false;
int accepted = 0;
int deferred = 0;
synchronized (pendingLock) {
for (String build : builds) {
if (build == null || build.isBlank() || build.equals(inFlight) || pending.contains(build)) continue;
if (pending.size() >= MAX_PENDING_BUILDS) {
deferred++;
continue;
}
pending.add(build);
accepted++;
}
if (!pending.isEmpty() && !drainScheduled) {
drainScheduled = true;
schedule = true;
}
}
if (deferred > 0) {
LOG.warn("Source filter index queue is full; {} builds keep raw-value fallback", deferred);
}
if (schedule) scheduleDrain();
return accepted;
}
private void scheduleDrain() {
try {
worker.execute(this::drainPending);
} catch (RejectedExecutionException failure) {
int queued;
synchronized (pendingLock) {
drainScheduled = false;
queued = pending.size();
}
LOG.error("Source filter index drain could not be scheduled; {} builds keep raw-value fallback", queued, failure);
}
}
private void drainPending() {
long started = System.nanoTime();
int succeeded = 0;
int failed = 0;
while (true) {
String buildId;
synchronized (pendingLock) {
if (pending.isEmpty()) {
drainScheduled = false;
break;
}
var iterator = pending.iterator();
buildId = iterator.next();
iterator.remove();
inFlight = buildId;
}
if (Thread.currentThread().isInterrupted()) {
synchronized (pendingLock) {
pending.add(buildId);
inFlight = null;
drainScheduled = false;
}
LOG.info("Source filter index drain interrupted; {} builds remain queued", pendingCount());
return;
}
long buildStarted = System.nanoTime();
try {
index.ensureBuild(buildId);
succeeded++;
LOG.info("Source filter index build completed: build={}, queuedRemaining={}, elapsedMs={}", buildId,
pendingCount(), (System.nanoTime() - buildStarted) / 1_000_000L);
} catch (RuntimeException failure) {
if (Thread.currentThread().isInterrupted()) {
synchronized (pendingLock) {
pending.add(buildId);
inFlight = null;
drainScheduled = false;
}
LOG.info("Source filter index drain interrupted while building {}; {} builds remain queued", buildId,
pendingCount());
return;
}
failed++;
LOG.error("Source filter index build failed: build={}; raw-value fallback remains available, continuing",
buildId, failure);
} finally {
synchronized (pendingLock) { inFlight = null; }
}
}
LOG.info("Source filter index drain finished: succeeded={}, failed={}, elapsedMs={}", succeeded, failed,
(System.nanoTime() - started) / 1_000_000L);
}
private int pendingCount() {
synchronized (pendingLock) { return pending.size(); }
}
@PreDestroy
public void stopWorker() {
worker.shutdownNow();
try {
if (!worker.awaitTermination(2, TimeUnit.SECONDS)) {
LOG.warn("Source filter index backfill worker did not stop within two seconds");
}
} catch (InterruptedException interrupted) {
Thread.currentThread().interrupt();
}
}
private static ExecutorService createWorker() {
return new ThreadPoolExecutor(1, 1, 0L, TimeUnit.MILLISECONDS, new ArrayBlockingQueue<>(1), runnable -> {
Thread thread = new Thread(runnable, "source-filter-index-backfill");
thread.setDaemon(true);
return thread;
}, new ThreadPoolExecutor.AbortPolicy());
}
}
@@ -5,15 +5,15 @@ import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.security.core.annotation.AuthenticationPrincipal; import org.springframework.security.core.annotation.AuthenticationPrincipal;
import org.springframework.web.bind.annotation.*; import org.springframework.web.bind.annotation.*;
@RestController @RequestMapping("/api/governance") @RestController @RequestMapping("/api/governance")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_PUBLISHER','DATA_MAINTAINER','RULE_MAINTAINER','AUDIT_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public class CandidateAcceptanceController { public class CandidateAcceptanceController {
private final CandidateAcceptanceService service; private final CandidateAcceptanceService service;
public CandidateAcceptanceController(CandidateAcceptanceService service){this.service=service;} public CandidateAcceptanceController(CandidateAcceptanceService service){this.service=service;}
@GetMapping("/migration-baseline") public Map<String,Object> baseline(){return service.migrationBaseline();} @GetMapping("/migration-baseline") public Map<String,Object> baseline(){return service.migrationBaseline();}
@GetMapping("/releases/{id}/reconciliations") public Map<String,Object> list(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor){return service.list(id,actor);} @GetMapping("/releases/{id}/reconciliations") public Map<String,Object> list(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor){return service.list(id,actor);}
@GetMapping("/reconciliations/{id}") public Map<String,Object> detail(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor){return service.detail(id,actor);} @GetMapping("/reconciliations/{id}") public Map<String,Object> detail(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor){return service.detail(id,actor);}
@PostMapping("/releases/{id}/reconciliations") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_PUBLISHER','DATA_MAINTAINER','RULE_MAINTAINER')") @PostMapping("/releases/{id}/reconciliations") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> create(@PathVariable String id,@RequestBody CandidateAcceptanceService.ReconciliationRequest request,@AuthenticationPrincipal AppUserPrincipal actor){return service.create(id,request,actor);} public Map<String,Object> create(@PathVariable String id,@RequestBody CandidateAcceptanceService.ReconciliationRequest request,@AuthenticationPrincipal AppUserPrincipal actor){return service.create(id,request,actor);}
@PostMapping("/reconciliations/{id}/confirm") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_PUBLISHER')") @PostMapping("/reconciliations/{id}/confirm") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> confirm(@PathVariable String id,@RequestBody CandidateAcceptanceService.Confirmation request,@AuthenticationPrincipal AppUserPrincipal actor){return service.confirm(id,request,actor);} public Map<String,Object> confirm(@PathVariable String id,@RequestBody CandidateAcceptanceService.Confirmation request,@AuthenticationPrincipal AppUserPrincipal actor){return service.confirm(id,request,actor);}
} }
@@ -32,7 +32,7 @@ public class CandidateAcceptanceService {
} }
} }
@Transactional public Map<String,Object> create(String releaseId,ReconciliationRequest request,AppUserPrincipal actor){ @Transactional public Map<String,Object> create(String releaseId,ReconciliationRequest request,AppUserPrincipal actor){
if(actor.getRoleCodes().stream().noneMatch(Set.of("SYSTEM_ADMIN","DATA_PUBLISHER","DATA_MAINTAINER","RULE_MAINTAINER")::contains))throw error(HttpStatus.FORBIDDEN,"当前角色不能执行候选验收"); if(actor.getRoleCodes().stream().noneMatch(Set.of("SYSTEM_ADMIN", "DATA_ADMIN")::contains))throw error(HttpStatus.FORBIDDEN,"当前角色不能执行候选验收");
requireRead(releaseId,actor);Map<String,Object> release=g.one("SELECT id,status,expected_current_release_id,complete,rule_version FROM data_release WHERE id=? FOR UPDATE",releaseId); requireRead(releaseId,actor);Map<String,Object> release=g.one("SELECT id,status,expected_current_release_id,complete,rule_version FROM data_release WHERE id=? FOR UPDATE",releaseId);
if(!"CANDIDATE".equals(release.get("status")))throw error(HttpStatus.CONFLICT,"只对候选版本创建基线验收"); if(!"CANDIDATE".equals(release.get("status")))throw error(HttpStatus.CONFLICT,"只对候选版本创建基线验收");
String baseline=request.baselineReleaseId()==null?(String)release.get("expectedCurrentReleaseId"):request.baselineReleaseId(); String baseline=request.baselineReleaseId()==null?(String)release.get("expectedCurrentReleaseId"):request.baselineReleaseId();
@@ -72,7 +72,7 @@ public class CandidateAcceptanceService {
public Map<String,Object> list(String releaseId,AppUserPrincipal actor){requireRead(releaseId,actor);return Map.of("items",g.rows("SELECT id,release_id,baseline_release_id,status,difference_hash,row_version,created_by,confirmed_by,confirmation_reason,created_at,confirmed_at FROM candidate_reconciliation WHERE release_id=? ORDER BY created_at DESC",releaseId),"acceptanceId",Objects.toString(g.jdbc.queryForObject("SELECT acceptance_id FROM data_release WHERE id=?",String.class,releaseId),""));} public Map<String,Object> list(String releaseId,AppUserPrincipal actor){requireRead(releaseId,actor);return Map.of("items",g.rows("SELECT id,release_id,baseline_release_id,status,difference_hash,row_version,created_by,confirmed_by,confirmation_reason,created_at,confirmed_at FROM candidate_reconciliation WHERE release_id=? ORDER BY created_at DESC",releaseId),"acceptanceId",Objects.toString(g.jdbc.queryForObject("SELECT acceptance_id FROM data_release WHERE id=?",String.class,releaseId),""));}
public Map<String,Object> detail(String id,AppUserPrincipal actor){Map<String,Object> row=g.one("SELECT * FROM candidate_reconciliation WHERE id=?",id);requireRead((String)row.get("releaseId"),actor);if(row.get("baselineReleaseId")!=null)requireRead((String)row.get("baselineReleaseId"),actor);requireTransitionRead(row,actor);row.put("request",decode(row.get("request")));row.put("publishReady",ready(row));return row;} public Map<String,Object> detail(String id,AppUserPrincipal actor){Map<String,Object> row=g.one("SELECT * FROM candidate_reconciliation WHERE id=?",id);requireRead((String)row.get("releaseId"),actor);if(row.get("baselineReleaseId")!=null)requireRead((String)row.get("baselineReleaseId"),actor);requireTransitionRead(row,actor);row.put("request",decode(row.get("request")));row.put("publishReady",ready(row));return row;}
@Transactional public Map<String,Object> confirm(String id,Confirmation request,AppUserPrincipal actor){ @Transactional public Map<String,Object> confirm(String id,Confirmation request,AppUserPrincipal actor){
if(!actor.getRoleCodes().contains("SYSTEM_ADMIN")&&!actor.getRoleCodes().contains("DATA_PUBLISHER"))throw error(HttpStatus.FORBIDDEN,"差异确认须由有范围权限的发布者执行"); if(!actor.getRoleCodes().contains("SYSTEM_ADMIN")&&!actor.getRoleCodes().contains("DATA_ADMIN"))throw error(HttpStatus.FORBIDDEN,"差异确认须由有范围权限的发布者执行");
Map<String,Object> row=g.one("SELECT * FROM candidate_reconciliation WHERE id=? FOR UPDATE",id);requireRead((String)row.get("releaseId"),actor);if(row.get("baselineReleaseId")!=null)requireRead((String)row.get("baselineReleaseId"),actor); Map<String,Object> row=g.one("SELECT * FROM candidate_reconciliation WHERE id=? FOR UPDATE",id);requireRead((String)row.get("releaseId"),actor);if(row.get("baselineReleaseId")!=null)requireRead((String)row.get("baselineReleaseId"),actor);
if(!"REVIEW_REQUIRED".equals(row.get("status"))||request.expectedRowVersion()==null||((Number)row.get("rowVersion")).longValue()!=request.expectedRowVersion()||!Objects.equals(row.get("differenceHash"),request.expectedDifferenceHash()))throw error(HttpStatus.CONFLICT,"仅可确认当前待审核的确切差异结果"); if(!"REVIEW_REQUIRED".equals(row.get("status"))||request.expectedRowVersion()==null||((Number)row.get("rowVersion")).longValue()!=request.expectedRowVersion()||!Objects.equals(row.get("differenceHash"),request.expectedDifferenceHash()))throw error(HttpStatus.CONFLICT,"仅可确认当前待审核的确切差异结果");
Map<String,Object> evidence=evidence(row);Object transitionHash=evidence.get("transitionDifferenceHash");if(transitionHash!=null&&!Objects.equals(transitionHash,request.expectedTransitionDifferenceHash()))throw error(HttpStatus.CONFLICT,"必须明确确认实际当前版本至候选的确切变更差异hash"); Map<String,Object> evidence=evidence(row);Object transitionHash=evidence.get("transitionDifferenceHash");if(transitionHash!=null&&!Objects.equals(transitionHash,request.expectedTransitionDifferenceHash()))throw error(HttpStatus.CONFLICT,"必须明确确认实际当前版本至候选的确切变更差异hash");
@@ -14,12 +14,15 @@ public class EffectiveAccessController {
Map<String,Object> out=new LinkedHashMap<>();out.put("userId",id);out.put("active",user.isEnabled());out.put("authorizationEffective",effective);out.put("roles",user.getRoleCodes());out.put("allowedActions",effective?actions:Set.of());out.put("departments",user.getRoleCodes().contains("SYSTEM_ADMIN")?List.of("*"):user.getDepartments());out.put("groups",user.getRoleCodes().contains("SYSTEM_ADMIN")?List.of("*"):user.getGroups());out.put("grantedDepartments",g.jdbc.query("SELECT scope_value FROM app_user_scope WHERE user_id=? AND scope_type='DEPARTMENT' ORDER BY scope_value",(rs,n)->rs.getString(1),id));out.put("hierarchySource",g.rows("SELECT c.release_id,r.mapping_version FROM current_release c JOIN data_release r ON r.id=c.release_id WHERE c.dataset_id='quality'"));out.put("fieldPolicy",policy);out.put("validUntil",policy.get("validUntil"));out.put("scopeHash",GovernanceSupport.sha(g.encode(List.of(user.getRoleCodes(),user.getDepartments(),user.getGroups(),policy))));return out;} Map<String,Object> out=new LinkedHashMap<>();out.put("userId",id);out.put("active",user.isEnabled());out.put("authorizationEffective",effective);out.put("roles",user.getRoleCodes());out.put("allowedActions",effective?actions:Set.of());out.put("departments",user.getRoleCodes().contains("SYSTEM_ADMIN")?List.of("*"):user.getDepartments());out.put("groups",user.getRoleCodes().contains("SYSTEM_ADMIN")?List.of("*"):user.getGroups());out.put("grantedDepartments",g.jdbc.query("SELECT scope_value FROM app_user_scope WHERE user_id=? AND scope_type='DEPARTMENT' ORDER BY scope_value",(rs,n)->rs.getString(1),id));out.put("hierarchySource",g.rows("SELECT c.release_id,r.mapping_version FROM current_release c JOIN data_release r ON r.id=c.release_id WHERE c.dataset_id='quality'"));out.put("fieldPolicy",policy);out.put("validUntil",policy.get("validUntil"));out.put("scopeHash",GovernanceSupport.sha(g.encode(List.of(user.getRoleCodes(),user.getDepartments(),user.getGroups(),policy))));return out;}
@GetMapping("/{id}/scope-preview")public Map<String,Object> preview(@PathVariable String id){AppUserPrincipal user=principal(id);Map<String,Object> effective=effective(id);String release=g.jdbc.queryForObject("SELECT release_id FROM current_release WHERE dataset_id='quality'",String.class);List<Map<String,Object>> counts=new ArrayList<>();for(String stage:List.of("black","bright")){String base="SELECT COUNT(*) FROM fact_"+stage+" f WHERE EXISTS(SELECT 1 FROM release_member rm WHERE rm.release_id=? AND rm.transform_build_id=f.build_id)";Long total=g.jdbc.queryForObject(base,Long.class,release);StringBuilder sql=new StringBuilder(base);List<Object> args=new ArrayList<>();args.add(release);DataScope.append(sql,args,user,"f.department","f.group_name");if(!Boolean.TRUE.equals(effective.get("authorizationEffective"))||!(effective.get("allowedActions") instanceof Collection<?> actions)||!actions.contains("query.aggregate"))sql.append(" AND 1=0");Long permitted=g.jdbc.queryForObject(sql.toString(),Long.class,args.toArray());counts.add(Map.of("stage",stage,"totalRows",total==null?0:total,"authorizedRows",permitted==null?0:permitted));}Map<String,Object> out=new LinkedHashMap<>(effective);out.put("releaseId",release);out.put("counts",counts);out.put("dataReturned",false);return out;} @GetMapping("/{id}/scope-preview")public Map<String,Object> preview(@PathVariable String id){AppUserPrincipal user=principal(id);Map<String,Object> effective=effective(id);String release=g.jdbc.queryForObject("SELECT release_id FROM current_release WHERE dataset_id='quality'",String.class);List<Map<String,Object>> counts=new ArrayList<>();for(String stage:List.of("black","bright")){String base="SELECT COUNT(*) FROM fact_"+stage+" f WHERE EXISTS(SELECT 1 FROM release_member rm WHERE rm.release_id=? AND rm.transform_build_id=f.build_id)";Long total=g.jdbc.queryForObject(base,Long.class,release);StringBuilder sql=new StringBuilder(base);List<Object> args=new ArrayList<>();args.add(release);DataScope.append(sql,args,user,"f.department","f.group_name");if(!Boolean.TRUE.equals(effective.get("authorizationEffective"))||!(effective.get("allowedActions") instanceof Collection<?> actions)||!actions.contains("query.aggregate"))sql.append(" AND 1=0");Long permitted=g.jdbc.queryForObject(sql.toString(),Long.class,args.toArray());counts.add(Map.of("stage",stage,"totalRows",total==null?0:total,"authorizedRows",permitted==null?0:permitted));}Map<String,Object> out=new LinkedHashMap<>(effective);out.put("releaseId",release);out.put("counts",counts);out.put("dataReturned",false);return out;}
private AppUserPrincipal principal(String id){String username=String.valueOf(g.one("SELECT username FROM app_user WHERE id=?",id).get("username"));return access.resolvePrincipal((AppUserPrincipal)users.loadUserByUsername(username));} private AppUserPrincipal principal(String id){String username=String.valueOf(g.one("SELECT username FROM app_user WHERE id=?",id).get("username"));return access.resolvePrincipal((AppUserPrincipal)users.loadUserByUsername(username));}
static Set<String> actions(List<String> roles){Set<String> out=new TreeSet<>();for(String role:roles)switch(role){ static Set<String> actions(List<String> roles) {
case "SYSTEM_ADMIN"->out.addAll(List.of("query.aggregate","query.detail","analysis.run","source.view","source.manage","source.receive","source.withdraw","source.evidence","source.download","import.run","import.retry","release.build","release.publish","release.rollback","rule.edit","rule.validate","rule.activate","mapping.edit","mapping.validate","mapping.activate","target.manage","cost.edit","task.view","task.retry","task.cancel","monitor.view","backup.manage","restore.isolated","iam.manage","service.manage","audit.view")); Set<String> out = new TreeSet<>();
case "QUALITY_VIEWER","BUSINESS_ANALYST"->out.addAll(List.of("query.aggregate","query.detail","analysis.run","source.view","source.evidence","source.download")); if (roles.size() != 1 || !cn.yunsheng.quality.identity.AppRoles.CODES.contains(roles.getFirst())) return out;
case "DATA_MAINTAINER"->out.addAll(List.of("source.view","source.manage","source.receive","source.withdraw","source.evidence","source.download","import.run","import.retry","mapping.edit","mapping.validate","mapping.activate","cost.edit","task.view","task.retry","task.cancel")); String role = roles.getFirst();
case "DATA_PUBLISHER"->out.addAll(List.of("source.view","source.evidence","source.download","release.build","release.publish","release.rollback","task.view")); out.addAll(List.of("query.aggregate", "query.detail", "analysis.run", "source.view", "source.evidence", "source.download", "rule.view", "mapping.view", "release.view", "metric.view", "inspection.run"));
case "RULE_MAINTAINER"->out.addAll(List.of("rule.edit","rule.validate","rule.activate","mapping.edit","mapping.validate","mapping.activate","target.manage","cost.edit")); if (Set.of("DATA_ADMIN", "SYSTEM_ADMIN").contains(role))
case "SYSTEM_OPERATOR"->out.addAll(List.of("task.view","task.cancel","monitor.view","backup.manage","restore.isolated")); out.addAll(List.of("source.manage", "source.receive", "source.withdraw", "import.run", "import.retry", "import.view", "release.build", "release.publish", "release.rollback", "mapping.edit", "mapping.validate", "mapping.activate", "target.manage", "cost.view", "cost.edit", "task.view", "task.retry", "task.cancel"));
case "AUDIT_VIEWER"->out.addAll(List.of("audit.view","source.evidence","source.download"));default->{}}if(roles.stream().anyMatch(Set.of("SYSTEM_ADMIN","QUALITY_VIEWER","BUSINESS_ANALYST","AUDIT_VIEWER")::contains))out.addAll(List.of("rule.view","mapping.view","release.view","metric.view"));if(roles.contains("RULE_MAINTAINER"))out.addAll(List.of("rule.view","mapping.view","cost.view","metric.view"));if(roles.contains("DATA_MAINTAINER"))out.addAll(List.of("mapping.view","cost.view","release.view","import.view"));if(roles.contains("DATA_PUBLISHER"))out.addAll(List.of("release.view","import.view"));return out;} if ("SYSTEM_ADMIN".equals(role))
out.addAll(List.of("rule.edit", "rule.validate", "rule.activate", "monitor.view", "backup.manage", "restore.isolated", "iam.manage", "service.manage", "audit.view", "inspection.manage"));
return out;
}
} }
@@ -19,6 +19,19 @@ public class GovernanceAccessService {
private final JsonMapper json; private final JsonMapper json;
public GovernanceAccessService(JdbcTemplate jdbc, JsonMapper json) { this.jdbc=jdbc; this.json=json; } public GovernanceAccessService(JdbcTemplate jdbc, JsonMapper json) { this.jdbc=jdbc; this.json=json; }
/** 角色改名沿用原指纹中的角色部分;实际角色改变后立即失效,不授予旧角色权限。 */
public List<String> fingerprintRoles(AppUserPrincipal user) {
List<String> current = user.getRoleCodes();
if (current.size() != 1 || !cn.yunsheng.quality.identity.AppRoles.CODES.contains(current.getFirst())) return current;
List<Map<String,Object>> anchors = jdbc.query("SELECT assigned_role,previous_roles FROM user_role_migration_anchor WHERE user_id=?",
(rs,n) -> Map.of("role",rs.getString(1),"previous",readList(rs.getString(2))),user.getId().toString());
if (anchors.isEmpty() || !current.getFirst().equals(anchors.getFirst().get("role"))) return current;
Object raw = anchors.getFirst().get("previous");
if (!(raw instanceof List<?> values) || values.isEmpty() || values.stream().anyMatch(v -> !(v instanceof String))) return current;
List<String> previous = values.stream().map(String.class::cast).distinct().sorted().toList();
return current.getFirst().equals(cn.yunsheng.quality.identity.AppRoles.migratedRole(previous)) ? previous : current;
}
public void requireDetail(AppUserPrincipal user) { public void requireDetail(AppUserPrincipal user) {
if (!Boolean.TRUE.equals(policy(user.getId().toString()).get("detailAllowed"))) denied("此身份仅允许汇总查询"); if (!Boolean.TRUE.equals(policy(user.getId().toString()).get("detailAllowed"))) denied("此身份仅允许汇总查询");
} }
@@ -3,7 +3,7 @@ import java.util.Map;
import org.springframework.security.access.prepost.PreAuthorize; import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.web.bind.annotation.*; import org.springframework.web.bind.annotation.*;
@RestController @RequestMapping("/api/governance") @RestController @RequestMapping("/api/governance")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_PUBLISHER','RULE_MAINTAINER','DATA_MAINTAINER','AUDIT_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public class GovernanceDependencyController { public class GovernanceDependencyController {
private final GovernanceDependencyService dependencies; private final GovernanceDependencyService dependencies;
private final GovernanceAccessService access; private final GovernanceAccessService access;
@@ -18,10 +18,10 @@ public class GovernanceIdentityController {
public GovernanceIdentityController(GovernanceSupport g,GovernanceAccessService access){this.g=g;this.access=access;} public GovernanceIdentityController(GovernanceSupport g,GovernanceAccessService access){this.g=g;this.access=access;}
@GetMapping("/users/{id}/access-policy") @PreAuthorize("hasRole('SYSTEM_ADMIN')") @GetMapping("/users/{id}/access-policy") @PreAuthorize("hasRole('SYSTEM_ADMIN')")
public Map<String,Object> policy(@PathVariable String id){g.one("SELECT id FROM app_user WHERE id=?",id);return Map.of("policy",access.policy(id));} public Map<String,Object> policy(@PathVariable String id){g.one("SELECT id FROM app_user WHERE id=? AND deleted_at IS NULL",id);return Map.of("policy",access.policy(id));}
@PutMapping("/users/{id}/access-policy") @PreAuthorize("hasRole('SYSTEM_ADMIN')") @Transactional @PutMapping("/users/{id}/access-policy") @PreAuthorize("hasRole('SYSTEM_ADMIN')") @Transactional
public Map<String,Object> policy(@PathVariable String id,@RequestBody PolicyRequest r,@AuthenticationPrincipal AppUserPrincipal actor){ public Map<String,Object> policy(@PathVariable String id,@RequestBody PolicyRequest r,@AuthenticationPrincipal AppUserPrincipal actor){
g.one("SELECT id FROM app_user WHERE id=? FOR UPDATE",id);g.required(r.reason(),"理由"); g.one("SELECT id FROM app_user WHERE id=? AND deleted_at IS NULL FOR UPDATE",id);g.required(r.reason(),"理由");
if(r.allowedFields()!=null&&r.allowedFields().stream().anyMatch(Objects::isNull))throw error(HttpStatus.BAD_REQUEST,"字段清单无效"); if(r.allowedFields()!=null&&r.allowedFields().stream().anyMatch(Objects::isNull))throw error(HttpStatus.BAD_REQUEST,"字段清单无效");
List<String> fields=r.allowedFields()==null?List.of():r.allowedFields().stream().distinct().sorted().toList(); List<String> fields=r.allowedFields()==null?List.of():r.allowedFields().stream().distinct().sorted().toList();
if(fields.size()>300||fields.stream().anyMatch(f->f==null||f.isBlank()||f.length()>190)||fields.contains("*")&&fields.size()!=1)throw error(HttpStatus.BAD_REQUEST,"字段清单无效"); if(fields.size()>300||fields.stream().anyMatch(f->f==null||f.isBlank()||f.length()>190)||fields.contains("*")&&fields.size()!=1)throw error(HttpStatus.BAD_REQUEST,"字段清单无效");
@@ -33,7 +33,7 @@ public class GovernanceIdentityController {
g.jdbc.update("UPDATE user_access_policy SET valid_until=?,inherit_department_children=? WHERE user_id=?",expiry,r.inheritDepartmentChildren(),id); g.jdbc.update("UPDATE user_access_policy SET valid_until=?,inherit_department_children=? WHERE user_id=?",expiry,r.inheritDepartmentChildren(),id);
g.audit(actor.getId().toString(),"ACCESS_POLICY_CHANGED","app_user",id,Map.of("before",before,"after",access.policy(id),"reason",r.reason()));return policy(id); g.audit(actor.getId().toString(),"ACCESS_POLICY_CHANGED","app_user",id,Map.of("before",before,"after",access.policy(id),"reason",r.reason()));return policy(id);
} }
@GetMapping("/service-identities") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR','AUDIT_VIEWER')") @GetMapping("/service-identities") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN')")
public Map<String,Object> identities(){return Map.of("items",g.rows("SELECT id,name,service_type,allowed_actions,credential_reference,credential_status,active,row_version,created_at,updated_at FROM service_identity ORDER BY name"),"allowedActions",ACTIONS,"serviceTypes",TYPE_ACTIONS.keySet());} public Map<String,Object> identities(){return Map.of("items",g.rows("SELECT id,name,service_type,allowed_actions,credential_reference,credential_status,active,row_version,created_at,updated_at FROM service_identity ORDER BY name"),"allowedActions",ACTIONS,"serviceTypes",TYPE_ACTIONS.keySet());}
@PostMapping("/service-identities") @PreAuthorize("hasRole('SYSTEM_ADMIN')") @Transactional @PostMapping("/service-identities") @PreAuthorize("hasRole('SYSTEM_ADMIN')") @Transactional
public Map<String,Object> create(@RequestBody IdentityRequest r,@AuthenticationPrincipal AppUserPrincipal actor){ public Map<String,Object> create(@RequestBody IdentityRequest r,@AuthenticationPrincipal AppUserPrincipal actor){
@@ -54,11 +54,11 @@ public class GovernanceIdentityController {
lock(id,r.expectedRowVersion());reference(r.credentialReference(),r.credentialStatus());g.required(r.reason(),"轮换理由"); lock(id,r.expectedRowVersion());reference(r.credentialReference(),r.credentialStatus());g.required(r.reason(),"轮换理由");
g.jdbc.update("UPDATE service_identity SET credential_reference=?,credential_status=?,active=IF(?='READY',active,FALSE),row_version=row_version+1 WHERE id=?",r.credentialReference(),r.credentialStatus(),r.credentialStatus(),id);rotation(id,r.credentialReference(),r.credentialStatus(),r.reason(),actor);return Map.of("identity",identity(id)); g.jdbc.update("UPDATE service_identity SET credential_reference=?,credential_status=?,active=IF(?='READY',active,FALSE),row_version=row_version+1 WHERE id=?",r.credentialReference(),r.credentialStatus(),r.credentialStatus(),id);rotation(id,r.credentialReference(),r.credentialStatus(),r.reason(),actor);return Map.of("identity",identity(id));
} }
@GetMapping("/service-identities/{id}/rotations") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','AUDIT_VIEWER')") @GetMapping("/service-identities/{id}/rotations") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN')")
public Map<String,Object> rotations(@PathVariable String id){return Map.of("items",g.rows("SELECT id,credential_reference,credential_status,reason,changed_by,created_at FROM service_credential_rotation WHERE service_identity_id=? ORDER BY id DESC",id));} public Map<String,Object> rotations(@PathVariable String id){return Map.of("items",g.rows("SELECT id,credential_reference,credential_status,reason,changed_by,created_at FROM service_credential_rotation WHERE service_identity_id=? ORDER BY id DESC",id));}
@PostMapping("/service-identities/{id}/authorize") @PreAuthorize("hasRole('SYSTEM_ADMIN')") @PostMapping("/service-identities/{id}/authorize") @PreAuthorize("hasRole('SYSTEM_ADMIN')")
public Map<String,Object> authorize(@PathVariable String id,@RequestBody Map<String,String> r){access.requireServiceAction(id,r.get("action"));return Map.of("authorized",true,"identityId",id,"action",r.get("action"));} public Map<String,Object> authorize(@PathVariable String id,@RequestBody Map<String,String> r){access.requireServiceAction(id,r.get("action"));return Map.of("authorized",true,"identityId",id,"action",r.get("action"));}
@GetMapping("/service-bindings") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR')") @GetMapping("/service-bindings") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN')")
public Map<String,Object> bindings(){return Map.of("items",g.rows("SELECT service_type,service_identity_id,updated_at FROM service_binding"));} public Map<String,Object> bindings(){return Map.of("items",g.rows("SELECT service_type,service_identity_id,updated_at FROM service_binding"));}
@PutMapping("/service-bindings/{type}") @PreAuthorize("hasRole('SYSTEM_ADMIN')") @Transactional @PutMapping("/service-bindings/{type}") @PreAuthorize("hasRole('SYSTEM_ADMIN')") @Transactional
public Map<String,Object> bind(@PathVariable String type,@RequestBody Map<String,String> r,@AuthenticationPrincipal AppUserPrincipal actor){ public Map<String,Object> bind(@PathVariable String type,@RequestBody Map<String,String> r,@AuthenticationPrincipal AppUserPrincipal actor){
@@ -8,7 +8,7 @@ import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.bind.annotation.*; import org.springframework.web.bind.annotation.*;
import static cn.yunsheng.quality.governance.GovernanceSupport.error; import static cn.yunsheng.quality.governance.GovernanceSupport.error;
@RestController @RequestMapping("/api/governance") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR')") @RestController @RequestMapping("/api/governance") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN')")
public class GovernanceOperationsController { public class GovernanceOperationsController {
private final GovernanceOperationsService service;private final GovernanceSupport g; private final GovernanceOperationsService service;private final GovernanceSupport g;
public GovernanceOperationsController(GovernanceOperationsService service,GovernanceSupport g){this.service=service;this.g=g;} public GovernanceOperationsController(GovernanceOperationsService service,GovernanceSupport g){this.service=service;this.g=g;}
@@ -24,7 +24,7 @@ public class GovernanceOperationsController {
g.jdbc.update("UPDATE monitor_alert_policy SET queue_age_minutes=?,disk_free_bytes=?,backup_age_hours=?,quiet_period_minutes=?,severity_overrides=CAST(? AS JSON),silence_until=?,slow_request_millis=?,row_version=row_version+1,updated_by=? WHERE id='default'",r.queueAgeMinutes(),r.diskFreeBytes(),r.backupAgeHours(),r.quietPeriodMinutes(),g.encode(r.severityOverrides()),silence,slow,actor.getId().toString());g.audit(actor.getId().toString(),"MONITOR_ALERT_POLICY_CHANGED","monitor_alert_policy","default",Map.of("reason",r.reason(),"before",before,"after",service.alertPolicy()));return alertPolicy();} g.jdbc.update("UPDATE monitor_alert_policy SET queue_age_minutes=?,disk_free_bytes=?,backup_age_hours=?,quiet_period_minutes=?,severity_overrides=CAST(? AS JSON),silence_until=?,slow_request_millis=?,row_version=row_version+1,updated_by=? WHERE id='default'",r.queueAgeMinutes(),r.diskFreeBytes(),r.backupAgeHours(),r.quietPeriodMinutes(),g.encode(r.severityOverrides()),silence,slow,actor.getId().toString());g.audit(actor.getId().toString(),"MONITOR_ALERT_POLICY_CHANGED","monitor_alert_policy","default",Map.of("reason",r.reason(),"before",before,"after",service.alertPolicy()));return alertPolicy();}
@GetMapping("/backups")public Map<String,Object> backups(){return Map.of("items",service.backups(),"plan",g.one("SELECT * FROM backup_plan WHERE id='default'"),"liveRestoreEnabled",false);} @GetMapping("/backups")public Map<String,Object> backups(){return Map.of("items",service.backups(),"plan",g.one("SELECT * FROM backup_plan WHERE id='default'"),"liveRestoreEnabled",false);}
@GetMapping("/backups/latest") @GetMapping("/backups/latest")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR','DATA_MAINTAINER','DATA_PUBLISHER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> latestBackup(){ public Map<String,Object> latestBackup(){
Map<String,Object> latest=service.backups().stream().max(Comparator.comparing(b->String.valueOf(b.get("createdAt")))).orElse(Map.of()); Map<String,Object> latest=service.backups().stream().max(Comparator.comparing(b->String.valueOf(b.get("createdAt")))).orElse(Map.of());
Map<String,Object> safe=new LinkedHashMap<>(); Map<String,Object> safe=new LinkedHashMap<>();
@@ -77,7 +77,7 @@ public class GovernanceOperationsService {
g.jdbc.update("UPDATE task SET status='FAILED',stage='FAILED',error_code=?,error_message=?,result=CAST(? AS JSON),retry_count=retry_count+1,next_retry_at=IF(retry_count<max_retries,TIMESTAMPADD(SECOND,30*POW(2,retry_count),CURRENT_TIMESTAMP(6)),NULL),lease_expires_at=NULL WHERE id=? AND lease_token=?",failureCode,failureMessage,g.encode(details),id,token); g.jdbc.update("UPDATE task SET status='FAILED',stage='FAILED',error_code=?,error_message=?,result=CAST(? AS JSON),retry_count=retry_count+1,next_retry_at=IF(retry_count<max_retries,TIMESTAMPADD(SECOND,30*POW(2,retry_count),CURRENT_TIMESTAMP(6)),NULL),lease_expires_at=NULL WHERE id=? AND lease_token=?",failureCode,failureMessage,g.encode(details),id,token);
} }
} }
private void requireOperator(String user){Integer count=g.jdbc.queryForObject("SELECT COUNT(*) FROM app_user u JOIN app_user_role r ON r.user_id=u.id WHERE u.id=? AND u.active=TRUE AND r.role_code IN('SYSTEM_ADMIN','SYSTEM_OPERATOR') AND NOT EXISTS(SELECT 1 FROM user_access_policy p WHERE p.user_id=u.id AND p.valid_until<CURRENT_TIMESTAMP(6))",Integer.class,user);if(count==null||count==0)throw error(HttpStatus.FORBIDDEN,"运维授权失效");} private void requireOperator(String user){Integer count=g.jdbc.queryForObject("SELECT COUNT(*) FROM app_user u JOIN app_user_role r ON r.user_id=u.id WHERE u.id=? AND u.active=TRUE AND r.role_code IN('SYSTEM_ADMIN') AND NOT EXISTS(SELECT 1 FROM user_access_policy p WHERE p.user_id=u.id AND p.valid_until<CURRENT_TIMESTAMP(6))",Integer.class,user);if(count==null||count==0)throw error(HttpStatus.FORBIDDEN,"运维授权失效");}
private Map<String,Object> runBackup(String type,Map<String,Object> payload)throws Exception{ private Map<String,Object> runBackup(String type,Map<String,Object> payload)throws Exception{
Path script=root.resolve("scripts/backup.py").toRealPath();if(!script.startsWith(root)||!Files.isRegularFile(script))throw new IllegalStateException("Fixed backup script missing"); Path script=root.resolve("scripts/backup.py").toRealPath();if(!script.startsWith(root)||!Files.isRegularFile(script))throw new IllegalStateException("Fixed backup script missing");
List<String> command=new ArrayList<>(List.of("python3",script.toString(),"BACKUP_VERIFY".equals(type)?"verify":"create-and-verify"));Path selected=null; List<String> command=new ArrayList<>(List.of("python3",script.toString(),"BACKUP_VERIFY".equals(type)?"verify":"create-and-verify"));Path selected=null;
@@ -5,7 +5,7 @@ import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.security.core.annotation.AuthenticationPrincipal; import org.springframework.security.core.annotation.AuthenticationPrincipal;
import org.springframework.web.bind.annotation.*; import org.springframework.web.bind.annotation.*;
@RestController @RequestMapping("/api/governance/mapping-issues") @RestController @RequestMapping("/api/governance/mapping-issues")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_PUBLISHER','DATA_MAINTAINER','RULE_MAINTAINER','AUDIT_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public class MappingIssueController { public class MappingIssueController {
private final MappingIssueService service; private final MappingIssueService service;
private final UnclassifiedSourceService unclassified; private final UnclassifiedSourceService unclassified;
@@ -19,8 +19,8 @@ public class MappingIssueController {
@GetMapping("/unclassified-projects") public Map<String,Object> unclassifiedProjects(@RequestParam(defaultValue="1") int page,@RequestParam(defaultValue="15") int pageSize,@RequestParam(required=false) String customerCode,@RequestParam(required=false) String businessGroup,@RequestParam(required=false) String department,@AuthenticationPrincipal AppUserPrincipal actor){return unclassified.listProjects(page,pageSize,customerCode,businessGroup,department,actor);} @GetMapping("/unclassified-projects") public Map<String,Object> unclassifiedProjects(@RequestParam(defaultValue="1") int page,@RequestParam(defaultValue="15") int pageSize,@RequestParam(required=false) String customerCode,@RequestParam(required=false) String businessGroup,@RequestParam(required=false) String department,@AuthenticationPrincipal AppUserPrincipal actor){return unclassified.listProjects(page,pageSize,customerCode,businessGroup,department,actor);}
@GetMapping public Map<String,Object> list(@RequestParam(required=false) String importRunId,@RequestParam(required=false) String status,@RequestParam(defaultValue="1") int page,@RequestParam(defaultValue="50") int pageSize,@RequestParam(required=false) String mappingType,@RequestParam(required=false) String customerCode,@RequestParam(required=false) String businessGroup,@RequestParam(required=false) String department,@AuthenticationPrincipal AppUserPrincipal actor){return service.list(importRunId,status,page,pageSize,customerCode,businessGroup,department,mappingType,actor);} @GetMapping public Map<String,Object> list(@RequestParam(required=false) String importRunId,@RequestParam(required=false) String status,@RequestParam(defaultValue="1") int page,@RequestParam(defaultValue="50") int pageSize,@RequestParam(required=false) String mappingType,@RequestParam(required=false) String customerCode,@RequestParam(required=false) String businessGroup,@RequestParam(required=false) String department,@AuthenticationPrincipal AppUserPrincipal actor){return service.list(importRunId,status,page,pageSize,customerCode,businessGroup,department,mappingType,actor);}
@GetMapping("/{id}") public Map<String,Object> detail(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor){return service.detail(id,actor);} @GetMapping("/{id}") public Map<String,Object> detail(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor){return service.detail(id,actor);}
@PostMapping("/{id}/resolve") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','RULE_MAINTAINER')") @PostMapping("/{id}/resolve") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> resolve(@PathVariable String id,@RequestBody MappingIssueService.Resolution request,@AuthenticationPrincipal AppUserPrincipal actor){return service.resolve(id,request,actor);} public Map<String,Object> resolve(@PathVariable String id,@RequestBody MappingIssueService.Resolution request,@AuthenticationPrincipal AppUserPrincipal actor){return service.resolve(id,request,actor);}
@PostMapping("/{id}/revalidate") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','DATA_PUBLISHER')") @PostMapping("/{id}/revalidate") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> revalidate(@PathVariable String id,@RequestBody MappingIssueService.Revalidation request,@AuthenticationPrincipal AppUserPrincipal actor){return service.revalidate(id,request,actor);} public Map<String,Object> revalidate(@PathVariable String id,@RequestBody MappingIssueService.Revalidation request,@AuthenticationPrincipal AppUserPrincipal actor){return service.revalidate(id,request,actor);}
} }
@@ -5,7 +5,7 @@ import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.security.core.annotation.AuthenticationPrincipal; import org.springframework.security.core.annotation.AuthenticationPrincipal;
import org.springframework.web.bind.annotation.*; import org.springframework.web.bind.annotation.*;
@RestController @RequestMapping("/api/governance") @RestController @RequestMapping("/api/governance")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','RULE_MAINTAINER','DATA_PUBLISHER','AUDIT_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public class RuleRegressionController { public class RuleRegressionController {
private final RuleRegressionService service; private final RuleRegressionService service;
public RuleRegressionController(RuleRegressionService service){this.service=service;} public RuleRegressionController(RuleRegressionService service){this.service=service;}
@@ -19,8 +19,18 @@ import org.springframework.stereotype.Service;
} }
static boolean overlaps(Coverage a,Coverage b) { static boolean overlaps(Coverage a,Coverage b) {
if(!Objects.equals(a.stage(),b.stage())||a.year()!=b.year()||a.month()!=b.month()||!SourceIdentity.intersects(a.scope(),b.scope()))return false; if(!Objects.equals(a.stage(),b.stage())||a.year()!=b.year()||a.month()!=b.month()||!SourceIdentity.intersects(a.scope(),b.scope()))return false;
return Objects.equals(SourceIdentity.canonicalSlot(a.slot()),SourceIdentity.canonicalSlot(b.slot())) if(Objects.equals(SourceIdentity.canonicalSlot(a.slot()),SourceIdentity.canonicalSlot(b.slot()))||Objects.equals(a.sha(),b.sha()))return true;
||!"SHARD_SNAPSHOT".equals(a.mode())||!"SHARD_SNAPSHOT".equals(b.mode())||Objects.equals(a.sha(),b.sha()); if("SHARD_SNAPSHOT".equals(a.mode())&&"SHARD_SNAPSHOT".equals(b.mode()))return false;
// APPEND_SNAPSHOT is created only by the explicit upload workflow. Its scope must match
// exactly before it can augment a complete monthly file; intersecting declarations still conflict.
if(isUploadAppend(a.mode())||isUploadAppend(b.mode())){
boolean supportedPair=(isUploadAppend(a.mode())&&isUploadAppend(b.mode()))
||("MONTHLY_SNAPSHOT".equals(a.mode())&&isUploadAppend(b.mode()))
||("MONTHLY_SNAPSHOT".equals(b.mode())&&isUploadAppend(a.mode()));
return !supportedPair||!Objects.equals(SourceIdentity.scope(a.scope()),SourceIdentity.scope(b.scope()));
}
return true;
} }
private static boolean isUploadAppend(String mode){return "APPEND_SNAPSHOT".equals(mode);}
record Coverage(String stage,int year,int month,String scope,String mode,String sha,String slot) {} record Coverage(String stage,int year,int month,String scope,String mode,String sha,String slot) {}
} }
@@ -16,13 +16,13 @@ import static cn.yunsheng.quality.governance.GovernanceSupport.error;
public class SourceGovernanceController { public class SourceGovernanceController {
private final SourceGovernanceService service;private final GovernanceSupport g;private final GovernanceAccessService access;private final SourceScanner scanner; private final SourceGovernanceService service;private final GovernanceSupport g;private final GovernanceAccessService access;private final SourceScanner scanner;
public SourceGovernanceController(SourceGovernanceService service,GovernanceSupport g,GovernanceAccessService access,SourceScanner scanner){this.service=service;this.g=g;this.access=access;this.scanner=scanner;} public SourceGovernanceController(SourceGovernanceService service,GovernanceSupport g,GovernanceAccessService access,SourceScanner scanner){this.service=service;this.g=g;this.access=access;this.scanner=scanner;}
@GetMapping("/protocols") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','DATA_PUBLISHER','SYSTEM_OPERATOR')") @GetMapping("/protocols") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> protocols(){return service.protocols();} public Map<String,Object> protocols(){return service.protocols();}
@PostMapping("/protocols") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER')") @PostMapping("/protocols") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> protocol(@RequestBody SourceGovernanceService.ProtocolRequest r,@AuthenticationPrincipal AppUserPrincipal actor){return service.protocol(r,actor);} public Map<String,Object> protocol(@RequestBody SourceGovernanceService.ProtocolRequest r,@AuthenticationPrincipal AppUserPrincipal actor){return service.protocol(r,actor);}
@PostMapping("/sources/{id}/resolve") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER')") @PostMapping("/sources/{id}/resolve") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> resolve(@PathVariable String id,@RequestBody SourceGovernanceService.ResolveRequest r,@AuthenticationPrincipal AppUserPrincipal actor){return service.resolve(id,r,actor);} public Map<String,Object> resolve(@PathVariable String id,@RequestBody SourceGovernanceService.ResolveRequest r,@AuthenticationPrincipal AppUserPrincipal actor){return service.resolve(id,r,actor);}
@GetMapping("/sources/{id}/identity-preview") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','DATA_PUBLISHER')") @GetMapping("/sources/{id}/identity-preview") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> identityPreview(@PathVariable String id,@RequestParam(required=false) String protocolId){ public Map<String,Object> identityPreview(@PathVariable String id,@RequestParam(required=false) String protocolId){
Map<String,Object> candidate=g.one("SELECT id,source_file,sha256,file_size,status,protocol_id FROM source_candidate WHERE id=?",id);String selected=protocolId==null?(String)candidate.get("protocolId"):protocolId; Map<String,Object> candidate=g.one("SELECT id,source_file,sha256,file_size,status,protocol_id FROM source_candidate WHERE id=?",id);String selected=protocolId==null?(String)candidate.get("protocolId"):protocolId;
if(selected==null)return Map.of("candidate",candidate,"protocol",Map.of(),"overlaps",List.of(),"decisionRequired",true,"allowedDecisions",List.of("RETURN","REPLACE","LEGAL_SHARD")); if(selected==null)return Map.of("candidate",candidate,"protocol",Map.of(),"overlaps",List.of(),"decisionRequired",true,"allowedDecisions",List.of("RETURN","REPLACE","LEGAL_SHARD"));
@@ -30,23 +30,23 @@ public class SourceGovernanceController {
List<Map<String,Object>> overlaps=g.rows("SELECT d.id AS source_document_id,d.source_slot,d.source_mode,d.scope_code,r.id AS source_revision_id,r.sha256,r.source_file FROM release_member m JOIN source_document d ON d.id=m.source_document_id JOIN source_revision r ON r.id=m.source_revision_id WHERE m.release_id=(SELECT release_id FROM current_release WHERE dataset_id='quality') AND d.stage=? AND d.source_year=? AND d.source_month=?",protocol.get("stage"),protocol.get("sourceYear"),protocol.get("sourceMonth")).stream().filter(d->SourceIdentity.intersects((String)protocol.get("scopeCode"),(String)d.get("scopeCode"))).toList(); List<Map<String,Object>> overlaps=g.rows("SELECT d.id AS source_document_id,d.source_slot,d.source_mode,d.scope_code,r.id AS source_revision_id,r.sha256,r.source_file FROM release_member m JOIN source_document d ON d.id=m.source_document_id JOIN source_revision r ON r.id=m.source_revision_id WHERE m.release_id=(SELECT release_id FROM current_release WHERE dataset_id='quality') AND d.stage=? AND d.source_year=? AND d.source_month=?",protocol.get("stage"),protocol.get("sourceYear"),protocol.get("sourceMonth")).stream().filter(d->SourceIdentity.intersects((String)protocol.get("scopeCode"),(String)d.get("scopeCode"))).toList();
return Map.of("candidate",candidate,"protocol",protocol,"overlaps",overlaps,"decisionRequired",true,"sameContentHint",overlaps.stream().anyMatch(o->Objects.equals(o.get("sha256"),candidate.get("sha256"))),"allowedDecisions","SHARD_SNAPSHOT".equals(protocol.get("sourceMode"))?List.of("RETURN","REPLACE","LEGAL_SHARD"):List.of("RETURN","REPLACE")); return Map.of("candidate",candidate,"protocol",protocol,"overlaps",overlaps,"decisionRequired",true,"sameContentHint",overlaps.stream().anyMatch(o->Objects.equals(o.get("sha256"),candidate.get("sha256"))),"allowedDecisions","SHARD_SNAPSHOT".equals(protocol.get("sourceMode"))?List.of("RETURN","REPLACE","LEGAL_SHARD"):List.of("RETURN","REPLACE"));
} }
@GetMapping("/source-documents") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','DATA_PUBLISHER')") @GetMapping("/source-documents") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> documents(){return Map.of("items",g.rows("SELECT d.id,d.source_slot,d.stage,d.source_year,d.source_month,d.scope_code,d.source_file,d.source_mode,d.applicable,(SELECT COUNT(*) FROM source_revision r WHERE r.source_document_id=d.id) AS revision_count FROM source_document d ORDER BY d.source_year,d.source_month,d.stage"),"revisions",g.rows("SELECT r.id,r.source_document_id,r.sha256,r.source_file,r.file_size,r.status,r.received_at FROM source_revision r ORDER BY r.received_at DESC"));} public Map<String,Object> documents(){return Map.of("items",g.rows("SELECT d.id,d.source_slot,d.stage,d.source_year,d.source_month,d.scope_code,d.source_file,d.source_mode,d.applicable,(SELECT COUNT(*) FROM source_revision r WHERE r.source_document_id=d.id) AS revision_count FROM source_document d ORDER BY d.source_year,d.source_month,d.stage"),"revisions",g.rows("SELECT r.id,r.source_document_id,r.sha256,r.source_file,r.file_size,r.status,r.received_at FROM source_revision r ORDER BY r.received_at DESC"));}
@GetMapping("/coverage") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','DATA_PUBLISHER','SYSTEM_OPERATOR')") @GetMapping("/coverage") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> coverage(@RequestParam(required=false) String releaseId){return service.coverage(releaseId);} public Map<String,Object> coverage(@RequestParam(required=false) String releaseId){return service.coverage(releaseId);}
@GetMapping("/source-changes") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','DATA_PUBLISHER')") @GetMapping("/source-changes") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> changes(){return service.changes();} public Map<String,Object> changes(){return service.changes();}
@PostMapping("/source-changes") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER')") @PostMapping("/source-changes") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> request(@RequestBody SourceGovernanceService.ChangeRequest r,@AuthenticationPrincipal AppUserPrincipal actor){return service.request(r,actor);} public Map<String,Object> request(@RequestBody SourceGovernanceService.ChangeRequest r,@AuthenticationPrincipal AppUserPrincipal actor){return service.request(r,actor);}
@PostMapping("/source-changes/{id}/review") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_PUBLISHER')") @PostMapping("/source-changes/{id}/review") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> review(@PathVariable String id,@RequestBody SourceGovernanceService.ReviewRequest r,@AuthenticationPrincipal AppUserPrincipal actor){return service.review(id,r,actor);} public Map<String,Object> review(@PathVariable String id,@RequestBody SourceGovernanceService.ReviewRequest r,@AuthenticationPrincipal AppUserPrincipal actor){return service.review(id,r,actor);}
@PostMapping("/source-changes/{id}/build") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_PUBLISHER')") @PostMapping("/source-changes/{id}/build") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> build(@PathVariable String id,@RequestBody SourceGovernanceService.BuildRequest r,@AuthenticationPrincipal AppUserPrincipal actor){return service.build(id,r,actor);} public Map<String,Object> build(@PathVariable String id,@RequestBody SourceGovernanceService.BuildRequest r,@AuthenticationPrincipal AppUserPrincipal actor){return service.build(id,r,actor);}
@GetMapping("/source-revisions/{id}/evidence") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','DATA_PUBLISHER','BUSINESS_ANALYST','QUALITY_VIEWER','AUDIT_VIEWER')") @Transactional @GetMapping("/source-revisions/{id}/evidence") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')") @Transactional
public Map<String,Object> evidence(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor){access.requireEvidence(actor);authorizeRevision(id,actor);Map<String,Object> revision=g.one("SELECT id,source_document_id,sha256,source_file,file_size,status,parser_version,received_at FROM source_revision WHERE id=?",id); public Map<String,Object> evidence(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor){access.requireEvidence(actor);authorizeRevision(id,actor);Map<String,Object> revision=g.one("SELECT id,source_document_id,sha256,source_file,file_size,status,parser_version,received_at FROM source_revision WHERE id=?",id);
g.audit(actor.getId().toString(),"SOURCE_EVIDENCE_READ","source_revision",id,Map.of());return Map.of("revision",revision,"builds",g.rows("SELECT id,parser_version,rule_version,mapping_version,status,row_count,rejected_count,component_summary FROM transform_build WHERE source_revision_id=?",id));} g.audit(actor.getId().toString(),"SOURCE_EVIDENCE_READ","source_revision",id,Map.of());return Map.of("revision",revision,"builds",g.rows("SELECT id,parser_version,rule_version,mapping_version,status,row_count,rejected_count,component_summary FROM transform_build WHERE source_revision_id=?",id));}
@GetMapping("/source-revisions/{id}/download") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','DATA_PUBLISHER','BUSINESS_ANALYST','QUALITY_VIEWER','AUDIT_VIEWER')") @Transactional @GetMapping("/source-revisions/{id}/download") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')") @Transactional
public ResponseEntity<byte[]> download(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor){ public ResponseEntity<byte[]> download(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor){
access.requireOriginalDownload(actor);authorizeRevision(id,actor);Map<String,Object> row=g.one("SELECT r.source_file,r.archive_path,r.file_size,r.sha256,d.stage FROM source_revision r JOIN source_document d ON d.id=r.source_document_id WHERE r.id=?",id); access.requireOriginalDownload(actor);authorizeRevision(id,actor);Map<String,Object> row=g.one("SELECT r.source_file,r.archive_path,r.file_size,r.sha256,d.stage FROM source_revision r JOIN source_document d ON d.id=r.source_document_id WHERE r.id=?",id);
try{Path path=Path.of(String.valueOf(row.get("archivePath")));if(Files.isSymbolicLink(path)||!Files.isRegularFile(path,LinkOption.NOFOLLOW_LINKS)||!path.toRealPath().startsWith(scanner.archiveRoot()))throw error(HttpStatus.CONFLICT,"归档文件不完整或位置无效"); try{Path path=Path.of(String.valueOf(row.get("archivePath")));if(Files.isSymbolicLink(path)||!Files.isRegularFile(path,LinkOption.NOFOLLOW_LINKS)||!path.toRealPath().startsWith(scanner.archiveRoot()))throw error(HttpStatus.CONFLICT,"归档文件不完整或位置无效");
@@ -18,9 +18,9 @@ public class TaskRetryService {
@SuppressWarnings("unchecked") public Map<String,Object> retry(String taskId,String key,String expectedCurrent,AppUserPrincipal actor){ @SuppressWarnings("unchecked") public Map<String,Object> retry(String taskId,String key,String expectedCurrent,AppUserPrincipal actor){
g.required(key,"幂等键");String runId=tx.execute(status->{Map<String,Object> task=g.one("SELECT * FROM task WHERE id=? FOR UPDATE",taskId); g.required(key,"幂等键");String runId=tx.execute(status->{Map<String,Object> task=g.one("SELECT * FROM task WHERE id=? FOR UPDATE",taskId);
boolean background=Set.of("RELEASE_POST_PUBLISH","BACKUP_CREATE_VERIFY","BACKUP_VERIFY").contains(String.valueOf(task.get("taskType"))); boolean background=Set.of("RELEASE_POST_PUBLISH","BACKUP_CREATE_VERIFY","BACKUP_VERIFY").contains(String.valueOf(task.get("taskType")));
if(!actor.getRoleCodes().contains("SYSTEM_ADMIN")&&!(background&&actor.getRoleCodes().contains("SYSTEM_OPERATOR"))&&!Objects.equals(task.get("createdBy"),actor.getId().toString()))throw error(HttpStatus.FORBIDDEN,"只能重试本人任务"); if(!actor.getRoleCodes().contains("SYSTEM_ADMIN")&&!(background&&actor.getRoleCodes().contains("SYSTEM_ADMIN"))&&!Objects.equals(task.get("createdBy"),actor.getId().toString()))throw error(HttpStatus.FORBIDDEN,"只能重试本人任务");
if(background){ if(background){
if(actor.getRoleCodes().stream().noneMatch(Set.of("SYSTEM_ADMIN","SYSTEM_OPERATOR")::contains))throw error(HttpStatus.FORBIDDEN,"后台运维任务需要运维授权"); if(actor.getRoleCodes().stream().noneMatch(Set.of("SYSTEM_ADMIN")::contains))throw error(HttpStatus.FORBIDDEN,"后台运维任务需要运维授权");
String requestKey="BACKGROUND_RETRY:"+GovernanceSupport.sha(key);List<Map<String,Object>> prior=g.rows("SELECT id,retry_parent_id,created_by FROM task WHERE idempotency_key=?",requestKey);if(!prior.isEmpty()){if(!Objects.equals(prior.getFirst().get("retryParentId"),taskId)||!Objects.equals(prior.getFirst().get("createdBy"),actor.getId().toString()))throw error(HttpStatus.CONFLICT,"幂等键已被使用");return "task:"+prior.getFirst().get("id");} String requestKey="BACKGROUND_RETRY:"+GovernanceSupport.sha(key);List<Map<String,Object>> prior=g.rows("SELECT id,retry_parent_id,created_by FROM task WHERE idempotency_key=?",requestKey);if(!prior.isEmpty()){if(!Objects.equals(prior.getFirst().get("retryParentId"),taskId)||!Objects.equals(prior.getFirst().get("createdBy"),actor.getId().toString()))throw error(HttpStatus.CONFLICT,"幂等键已被使用");return "task:"+prior.getFirst().get("id");}
if(!"FAILED".equals(task.get("status")))throw error(HttpStatus.CONFLICT,"只有失败任务可以重试"); if(!"FAILED".equals(task.get("status")))throw error(HttpStatus.CONFLICT,"只有失败任务可以重试");
String rootId=task.get("retryRootId")==null?taskId:String.valueOf(task.get("retryRootId"));Map<String,Object> root=rootId.equals(taskId)?task:g.one("SELECT retry_count,max_retries FROM task WHERE id=? FOR UPDATE",rootId);int count=((Number)root.get("retryCount")).intValue(),max=((Number)root.get("maxRetries")).intValue();if(count>=max)throw error(HttpStatus.CONFLICT,"此后台任务链已达重试上限"); String rootId=task.get("retryRootId")==null?taskId:String.valueOf(task.get("retryRootId"));Map<String,Object> root=rootId.equals(taskId)?task:g.one("SELECT retry_count,max_retries FROM task WHERE id=? FOR UPDATE",rootId);int count=((Number)root.get("retryCount")).intValue(),max=((Number)root.get("maxRetries")).intValue();if(count>=max)throw error(HttpStatus.CONFLICT,"此后台任务链已达重试上限");
@@ -35,10 +35,7 @@ public class AdminBootstrapRunner implements ApplicationRunner {
UUID id = UUID.randomUUID(); UUID id = UUID.randomUUID();
jdbc.update("INSERT INTO app_user (id, username, display_name, password_hash, active) VALUES (?, ?, ?, ?, TRUE)", jdbc.update("INSERT INTO app_user (id, username, display_name, password_hash, active) VALUES (?, ?, ?, ?, TRUE)",
id.toString(), username, username, encoder.encode(password)); id.toString(), username, username, encoder.encode(password));
for (String role : new String[]{"SYSTEM_ADMIN", "QUALITY_VIEWER", "BUSINESS_ANALYST", "DATA_MAINTAINER", jdbc.update("INSERT INTO app_user_role (user_id, role_code) VALUES (?, 'SYSTEM_ADMIN')", id.toString());
"RULE_MAINTAINER", "DATA_PUBLISHER", "SYSTEM_OPERATOR", "AUDIT_VIEWER"}) {
jdbc.update("INSERT INTO app_user_role (user_id, role_code) VALUES (?, ?)", id.toString(), role);
}
jdbc.update("INSERT INTO app_user_scope (user_id, scope_type, scope_value) VALUES (?, 'DEPARTMENT', '质量中心')", id.toString()); jdbc.update("INSERT INTO app_user_scope (user_id, scope_type, scope_value) VALUES (?, 'DEPARTMENT', '质量中心')", id.toString());
} }
} }
@@ -0,0 +1,36 @@
package cn.yunsheng.quality.identity;
import java.util.List;
import java.util.Set;
import org.springframework.http.HttpStatus;
import org.springframework.security.core.authority.SimpleGrantedAuthority;
import org.springframework.web.server.ResponseStatusException;
/** Exactly one assigned role; higher roles inherit lower operation permissions, never data scopes. */
public final class AppRoles {
public static final Set<String> CODES = Set.of("USER", "DATA_ADMIN", "SYSTEM_ADMIN");
private AppRoles() {}
public static String migratedRole(List<String> previous) {
if (previous.contains("SYSTEM_ADMIN")) return "SYSTEM_ADMIN";
if (previous.stream().anyMatch(Set.of("DATA_ADMIN", "DATA_MAINTAINER", "DATA_PUBLISHER", "RULE_MAINTAINER")::contains)) return "DATA_ADMIN";
if (previous.stream().anyMatch(Set.of("USER", "QUALITY_VIEWER", "BUSINESS_ANALYST", "SYSTEM_OPERATOR", "AUDIT_VIEWER")::contains)) return "USER";
return "";
}
public static List<String> validate(List<String> roles) {
if (roles == null || roles.size() != 1 || roles.getFirst() == null || !CODES.contains(roles.getFirst()))
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "请选择一个角色:用户、数据管理员或系统管理员");
return List.copyOf(roles);
}
public static List<SimpleGrantedAuthority> authorities(List<String> roles) {
if (roles == null || roles.size() != 1 || roles.getFirst() == null || !CODES.contains(roles.getFirst())) return List.of();
List<String> inherited = switch (roles.getFirst()) {
case "SYSTEM_ADMIN" -> List.of("SYSTEM_ADMIN", "DATA_ADMIN", "USER");
case "DATA_ADMIN" -> List.of("DATA_ADMIN", "USER");
default -> List.of("USER");
};
return inherited.stream().map(role -> new SimpleGrantedAuthority("ROLE_" + role)).toList();
}
}
@@ -1,10 +1,8 @@
package cn.yunsheng.quality.identity; package cn.yunsheng.quality.identity;
import java.util.ArrayList;
import java.util.List; import java.util.List;
import java.util.UUID; import java.util.UUID;
import org.springframework.jdbc.core.JdbcTemplate; import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.security.core.authority.SimpleGrantedAuthority;
import org.springframework.security.core.userdetails.UserDetails; import org.springframework.security.core.userdetails.UserDetails;
import org.springframework.security.core.userdetails.UserDetailsService; import org.springframework.security.core.userdetails.UserDetailsService;
import org.springframework.security.core.userdetails.UsernameNotFoundException; import org.springframework.security.core.userdetails.UsernameNotFoundException;
@@ -32,7 +30,7 @@ public class JdbcAppUserDetailsService implements UserDetailsService {
(rs, row) -> rs.getString(1), user.id().toString()); (rs, row) -> rs.getString(1), user.id().toString());
List<String> departments = scopeValues(user.id(), "DEPARTMENT"); List<String> departments = scopeValues(user.id(), "DEPARTMENT");
List<String> groups = scopeValues(user.id(), "GROUP"); List<String> groups = scopeValues(user.id(), "GROUP");
var authorities = roles.stream().map(role -> new SimpleGrantedAuthority("ROLE_" + role)).toList(); var authorities = AppRoles.authorities(roles);
return new AppUserPrincipal(user.id(), user.username(), user.passwordHash(), user.displayName(), user.active(), return new AppUserPrincipal(user.id(), user.username(), user.passwordHash(), user.displayName(), user.active(),
user.forcePasswordChange(), roles, departments, groups, authorities); user.forcePasswordChange(), roles, departments, groups, authorities);
} }
@@ -21,7 +21,7 @@ public class ImportRunController {
public ImportRunController(ImportRunService service, ImportWorker worker) { this.service = service; this.worker = worker; } public ImportRunController(ImportRunService service, ImportWorker worker) { this.service = service; this.worker = worker; }
@PostMapping @PostMapping
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> create(@Valid @RequestBody ImportRunService.CreateImportRequest request, public Map<String, Object> create(@Valid @RequestBody ImportRunService.CreateImportRequest request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
try { try {
@@ -26,7 +26,7 @@ public class ImportRunReadController {
public ImportRunReadController(JdbcTemplate jdbc, JsonMapper json,cn.yunsheng.quality.governance.GovernanceAccessService access) { this.jdbc = jdbc; this.json = json;this.access=access; } public ImportRunReadController(JdbcTemplate jdbc, JsonMapper json,cn.yunsheng.quality.governance.GovernanceAccessService access) { this.jdbc = jdbc; this.json = json;this.access=access; }
@GetMapping("/{id}") @GetMapping("/{id}")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR','DATA_MAINTAINER','DATA_PUBLISHER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> get(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> get(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor) {
authorize(id,actor); authorize(id,actor);
List<Map<String, Object>> rows = jdbc.query(""" List<Map<String, Object>> rows = jdbc.query("""
@@ -46,7 +46,7 @@ public class ImportRunReadController {
} }
@GetMapping("/{id}/issues") @GetMapping("/{id}/issues")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','SYSTEM_OPERATOR','DATA_MAINTAINER','DATA_PUBLISHER','AUDIT_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> issues(@PathVariable String id, public Map<String, Object> issues(@PathVariable String id,
@RequestParam(defaultValue = "1") int page, @RequestParam(defaultValue = "1") int page,
@RequestParam(defaultValue = "50") int pageSize,@AuthenticationPrincipal AppUserPrincipal actor) { @RequestParam(defaultValue = "50") int pageSize,@AuthenticationPrincipal AppUserPrincipal actor) {
@@ -70,7 +70,7 @@ public class ImportRunReadController {
if(!access.hasFullFields(actor))items=items.stream().map(i->{i.remove("details");if(!access.allowedFields(actor).contains(String.valueOf(i.get("field")))){i.remove("field");i.put("message","字段受限:请联系有权维护者处理");}return i;}).toList(); if(!access.hasFullFields(actor))items=items.stream().map(i->{i.remove("details");if(!access.allowedFields(actor).contains(String.valueOf(i.get("field")))){i.remove("field");i.put("message","字段受限:请联系有权维护者处理");}return i;}).toList();
return Map.of("items", items, "total", total == null ? 0 : total, "page", page, "pageSize", pageSize); return Map.of("items", items, "total", total == null ? 0 : total, "page", page, "pageSize", pageSize);
} }
private void authorize(String id,AppUserPrincipal actor){List<String> owners=jdbc.query("SELECT created_by FROM import_run WHERE id=?",(rs,n)->rs.getString(1),id);if(owners.isEmpty())throw new ResponseStatusException(HttpStatus.NOT_FOUND,"导入任务不存在");if(!actor.getRoleCodes().contains("SYSTEM_ADMIN")&&!actor.getRoleCodes().contains("SYSTEM_OPERATOR")&&!java.util.Objects.equals(owners.getFirst(),actor.getId().toString()))throw new ResponseStatusException(HttpStatus.FORBIDDEN,"无权访问该导入任务");} private void authorize(String id,AppUserPrincipal actor){List<String> owners=jdbc.query("SELECT created_by FROM import_run WHERE id=?",(rs,n)->rs.getString(1),id);if(owners.isEmpty())throw new ResponseStatusException(HttpStatus.NOT_FOUND,"导入任务不存在");if(!actor.getRoleCodes().contains("SYSTEM_ADMIN")&&!java.util.Objects.equals(owners.getFirst(),actor.getId().toString()))throw new ResponseStatusException(HttpStatus.FORBIDDEN,"无权访问该导入任务");}
private Object readJson(String value) { private Object readJson(String value) {
if (value == null) return Map.of(); if (value == null) return Map.of();
@@ -53,7 +53,7 @@ public class ImportRunService {
private ImportRunView create(CreateImportRequest request, UUID actorId, int maximumSources) { private ImportRunView create(CreateImportRequest request, UUID actorId, int maximumSources) {
access.requireBoundAction("IMPORTER","import.run"); access.requireBoundAction("IMPORTER","import.run");
Integer active=jdbc.queryForObject("SELECT COUNT(*) FROM app_user u JOIN app_user_role r ON r.user_id=u.id WHERE u.id=? AND u.active=TRUE AND r.role_code IN('SYSTEM_ADMIN','DATA_MAINTAINER','DATA_PUBLISHER')",Integer.class,actorId.toString()); Integer active=jdbc.queryForObject("SELECT COUNT(*) FROM app_user u JOIN app_user_role r ON r.user_id=u.id WHERE u.id=? AND u.active=TRUE AND r.role_code IN('SYSTEM_ADMIN', 'DATA_ADMIN')",Integer.class,actorId.toString());
if(active==null||active==0)throw new org.springframework.web.server.ResponseStatusException(org.springframework.http.HttpStatus.FORBIDDEN,"当前身份已不再具有导入/候选构建权限"); if(active==null||active==0)throw new org.springframework.web.server.ResponseStatusException(org.springframework.http.HttpStatus.FORBIDDEN,"当前身份已不再具有导入/候选构建权限");
if (request.sourceIds() == null || request.sourceIds().isEmpty() || request.sourceIds().size() > maximumSources) { if (request.sourceIds() == null || request.sourceIds().isEmpty() || request.sourceIds().size() > maximumSources) {
throw new IllegalArgumentException("请选择 1 到 " + maximumSources + " 个数据源"); throw new IllegalArgumentException("请选择 1 到 " + maximumSources + " 个数据源");
@@ -137,7 +137,7 @@ public class ImportRunService {
List<Boolean> active=jdbc.query("SELECT active FROM app_user WHERE id=? FOR UPDATE",(rs,n)->rs.getBoolean(1),actorId); List<Boolean> active=jdbc.query("SELECT active FROM app_user WHERE id=? FOR UPDATE",(rs,n)->rs.getBoolean(1),actorId);
if(active.isEmpty()||!active.getFirst())throw new org.springframework.web.server.ResponseStatusException(org.springframework.http.HttpStatus.FORBIDDEN,"提交人已停用"); if(active.isEmpty()||!active.getFirst())throw new org.springframework.web.server.ResponseStatusException(org.springframework.http.HttpStatus.FORBIDDEN,"提交人已停用");
List<String> roles=jdbc.query("SELECT role_code FROM app_user_role WHERE user_id=? FOR UPDATE",(rs,n)->rs.getString(1),actorId); List<String> roles=jdbc.query("SELECT role_code FROM app_user_role WHERE user_id=? FOR UPDATE",(rs,n)->rs.getString(1),actorId);
if(roles.stream().noneMatch(java.util.Set.of("SYSTEM_ADMIN","DATA_MAINTAINER","DATA_PUBLISHER")::contains))throw new org.springframework.web.server.ResponseStatusException(org.springframework.http.HttpStatus.FORBIDDEN,"提交人已无导入权限"); if(roles.stream().noneMatch(java.util.Set.of("SYSTEM_ADMIN", "DATA_ADMIN")::contains))throw new org.springframework.web.server.ResponseStatusException(org.springframework.http.HttpStatus.FORBIDDEN,"提交人已无导入权限");
List<String> departments=jdbc.query("SELECT scope_value FROM app_user_scope WHERE user_id=? AND scope_type='DEPARTMENT' FOR UPDATE",(rs,n)->rs.getString(1),actorId); List<String> departments=jdbc.query("SELECT scope_value FROM app_user_scope WHERE user_id=? AND scope_type='DEPARTMENT' FOR UPDATE",(rs,n)->rs.getString(1),actorId);
List<String> groups=jdbc.query("SELECT scope_value FROM app_user_scope WHERE user_id=? AND scope_type='GROUP' FOR UPDATE",(rs,n)->rs.getString(1),actorId); List<String> groups=jdbc.query("SELECT scope_value FROM app_user_scope WHERE user_id=? AND scope_type='GROUP' FOR UPDATE",(rs,n)->rs.getString(1),actorId);
var actor=new cn.yunsheng.quality.identity.AppUserPrincipal(UUID.fromString(actorId),actorId,"","",true,false,roles,departments,groups,List.of()); var actor=new cn.yunsheng.quality.identity.AppUserPrincipal(UUID.fromString(actorId),actorId,"","",true,false,roles,departments,groups,List.of());
@@ -67,7 +67,7 @@ public class ImportWorker {
if (runs.isEmpty()) throw new IllegalArgumentException("导入任务不存在"); if (runs.isEmpty()) throw new IllegalArgumentException("导入任务不存在");
RunRow run = runs.getFirst(); RunRow run = runs.getFirst();
if (List.of("COMPLETED", "FAILED", "CANCELLED").contains(run.status())) return getRun(runId); if (List.of("COMPLETED", "FAILED", "CANCELLED").contains(run.status())) return getRun(runId);
Integer authorized=jdbc.queryForObject("SELECT COUNT(*) FROM app_user u JOIN app_user_role r ON r.user_id=u.id WHERE u.id=? AND u.active=TRUE AND r.role_code IN('SYSTEM_ADMIN','DATA_MAINTAINER','DATA_PUBLISHER')",Integer.class,run.createdBy()); Integer authorized=jdbc.queryForObject("SELECT COUNT(*) FROM app_user u JOIN app_user_role r ON r.user_id=u.id WHERE u.id=? AND u.active=TRUE AND r.role_code IN('SYSTEM_ADMIN', 'DATA_ADMIN')",Integer.class,run.createdBy());
if(authorized==null||authorized==0)return fail(runId,"AUTHORIZATION_REVOKED","提交人已停用或撤回导入权限,请由有效维护者重新提交"); if(authorized==null||authorized==0)return fail(runId,"AUTHORIZATION_REVOKED","提交人已停用或撤回导入权限,请由有效维护者重新提交");
String expectedPointer = run.expectedCurrentReleaseId(); String expectedPointer = run.expectedCurrentReleaseId();
String currentPointer = currentReleaseId(); String currentPointer = currentReleaseId();
@@ -255,8 +255,7 @@ public class ImportWorker {
issues.add(new Issue(candidate, row, "INVALID_NUMBER", field, "数值字段不能转换为重量数值,原值已保留在raw中", true)); issues.add(new Issue(candidate, row, "INVALID_NUMBER", field, "数值字段不能转换为重量数值,原值已保留在raw中", true));
} }
if (value.unmappedCategory()) { if (value.unmappedCategory()) {
blocking[0]++; issues.add(new Issue(candidate, row, "CATEGORY_MAPPING_MISSING", "PDT代码/销售片区", "未命中业务分组对应关系,已保留为未分组,可在对应关系中补充归类", false));
issues.add(new Issue(candidate, row, "CATEGORY_MAPPING_MISSING", "PDT代码/销售片区", "非空来源键未命中已验证的产业部-地区映射", true));
} else if ("source_key_blank".equals(value.components().get("categoryMappingStatus"))) { } else if ("source_key_blank".equals(value.components().get("categoryMappingStatus"))) {
blankCategory[0]++; blankCategory[0]++;
issues.add(new Issue(candidate, row, "SOURCE_CATEGORY_KEY_BLANK", "PDT代码/销售片区", "原始匹配键为空,按源模型趋势上下文默认排除;原始行保留", false)); issues.add(new Issue(candidate, row, "SOURCE_CATEGORY_KEY_BLANK", "PDT代码/销售片区", "原始匹配键为空,按源模型趋势上下文默认排除;原始行保留", false));
@@ -455,7 +454,7 @@ public class ImportWorker {
} }
private boolean authorizedToSubmit(String actor,List<BuildMember> members){ private boolean authorizedToSubmit(String actor,List<BuildMember> members){
List<Boolean> enabled=jdbc.query("SELECT active FROM app_user WHERE id=? FOR UPDATE",(rs,n)->rs.getBoolean(1),actor);if(enabled.isEmpty()||!enabled.getFirst())return false; List<Boolean> enabled=jdbc.query("SELECT active FROM app_user WHERE id=? FOR UPDATE",(rs,n)->rs.getBoolean(1),actor);if(enabled.isEmpty()||!enabled.getFirst())return false;
List<String> roles=jdbc.query("SELECT role_code FROM app_user_role WHERE user_id=? FOR UPDATE",(rs,n)->rs.getString(1),actor);if(roles.stream().noneMatch(Set.of("SYSTEM_ADMIN","DATA_MAINTAINER","DATA_PUBLISHER")::contains))return false; List<String> roles=jdbc.query("SELECT role_code FROM app_user_role WHERE user_id=? FOR UPDATE",(rs,n)->rs.getString(1),actor);if(roles.stream().noneMatch(Set.of("SYSTEM_ADMIN", "DATA_ADMIN")::contains))return false;
List<java.sql.Timestamp> expiry=jdbc.query("SELECT valid_until FROM user_access_policy WHERE user_id=? FOR UPDATE",(rs,n)->rs.getTimestamp(1),actor);if(!expiry.isEmpty()&&expiry.getFirst()!=null&&expiry.getFirst().toInstant().isBefore(Instant.now()))return false; List<java.sql.Timestamp> expiry=jdbc.query("SELECT valid_until FROM user_access_policy WHERE user_id=? FOR UPDATE",(rs,n)->rs.getTimestamp(1),actor);if(!expiry.isEmpty()&&expiry.getFirst()!=null&&expiry.getFirst().toInstant().isBefore(Instant.now()))return false;
List<String> departments=jdbc.query("SELECT scope_value FROM app_user_scope WHERE user_id=? AND scope_type='DEPARTMENT' FOR UPDATE",(rs,n)->rs.getString(1),actor),groups=jdbc.query("SELECT scope_value FROM app_user_scope WHERE user_id=? AND scope_type='GROUP' FOR UPDATE",(rs,n)->rs.getString(1),actor); List<String> departments=jdbc.query("SELECT scope_value FROM app_user_scope WHERE user_id=? AND scope_type='DEPARTMENT' FOR UPDATE",(rs,n)->rs.getString(1),actor),groups=jdbc.query("SELECT scope_value FROM app_user_scope WHERE user_id=? AND scope_type='GROUP' FOR UPDATE",(rs,n)->rs.getString(1),actor);
cn.yunsheng.quality.identity.AppUserPrincipal user=access.resolvePrincipal(new cn.yunsheng.quality.identity.AppUserPrincipal(UUID.fromString(actor),actor,"","",true,false,roles,departments,groups,List.of())); cn.yunsheng.quality.identity.AppUserPrincipal user=access.resolvePrincipal(new cn.yunsheng.quality.identity.AppUserPrincipal(UUID.fromString(actor),actor,"","",true,false,roles,departments,groups,List.of()));
@@ -19,6 +19,8 @@ final class TransformBuildKey {
Integer.toString(source.year()), Integer.toString(source.month()), parserVersion, Integer.toString(source.year()), Integer.toString(source.month()), parserVersion,
rulesVersion, mappingVersion); rulesVersion, mappingVersion);
if(parserConfigHash!=null&&!"legacy".equals(parserConfigHash))identity+="\n"+parserConfigHash; if(parserConfigHash!=null&&!"legacy".equals(parserConfigHash))identity+="\n"+parserConfigHash;
// Validation policy is part of cache identity; old mapping rejections must not survive a retry.
identity += "\nunassigned-mapping-nonblocking-v1";
try { try {
return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(identity.getBytes(StandardCharsets.UTF_8))); return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(identity.getBytes(StandardCharsets.UTF_8)));
} catch (NoSuchAlgorithmException e) { } catch (NoSuchAlgorithmException e) {
@@ -9,15 +9,15 @@ import org.springframework.web.multipart.MultipartFile;
@RestController @RestController
@RequestMapping("/api/master-data/workbooks") @RequestMapping("/api/master-data/workbooks")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','RULE_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public class MasterDataWorkbookController { public class MasterDataWorkbookController {
private final MasterDataWorkbookService service; private final MasterDataWorkbookService service;
public MasterDataWorkbookController(MasterDataWorkbookService service) { this.service = service; } public MasterDataWorkbookController(MasterDataWorkbookService service) { this.service = service; }
@PostMapping("/preview") @PostMapping("/preview")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','RULE_MAINTAINER','DATA_PUBLISHER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> preview(@RequestPart("file") MultipartFile file, @AuthenticationPrincipal AppUserPrincipal actor) { return service.preview(file, actor); } public Map<String, Object> preview(@RequestPart("file") MultipartFile file, @AuthenticationPrincipal AppUserPrincipal actor) { return service.preview(file, actor); }
@PostMapping("/prepare") @PostMapping("/prepare")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','RULE_MAINTAINER','DATA_PUBLISHER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> prepareWorkbook(@RequestBody MasterDataWorkbookService.ImportRequest request, @AuthenticationPrincipal AppUserPrincipal actor) { return service.prepareWorkbook(request, actor); } public Map<String, Object> prepareWorkbook(@RequestBody MasterDataWorkbookService.ImportRequest request, @AuthenticationPrincipal AppUserPrincipal actor) { return service.prepareWorkbook(request, actor); }
@PostMapping("/import") @PostMapping("/import")
public Map<String, Object> importWorkbook(@RequestBody MasterDataWorkbookService.ImportRequest request, @AuthenticationPrincipal AppUserPrincipal actor) { return service.importWorkbook(request, actor); } public Map<String, Object> importWorkbook(@RequestBody MasterDataWorkbookService.ImportRequest request, @AuthenticationPrincipal AppUserPrincipal actor) { return service.importWorkbook(request, actor); }
@@ -184,11 +184,15 @@ public class MasterDataWorkbookService {
line++; String code = text(source, columns, "code"), name = text(source, columns, "name"); line++; String code = text(source, columns, "code"), name = text(source, columns, "name");
if (code.isBlank() && !name.isBlank()) { if (code.isBlank() && !name.isBlank()) {
code = codesByName.getOrDefault(name, "REF-" + UUID.nameUUIDFromBytes((kind + ":" + name).getBytes(java.nio.charset.StandardCharsets.UTF_8))); code = codesByName.getOrDefault(name, "REF-" + UUID.nameUUIDFromBytes((kind + ":" + name).getBytes(java.nio.charset.StandardCharsets.UTF_8)));
if (existing.containsKey(code) && !name.equals(existing.get(code).get("name"))) throw bad("第" + line + "行自动编码与已有记录冲突,请指定编码列"); while ((existing.containsKey(code) && !name.equals(existing.get(code).get("name")))
|| imported.containsKey(code) && !name.equals(imported.get(code).get("name")))
code = "REF-" + UUID.randomUUID();
} }
if (code.isBlank() || name.isBlank() || code.length() > 100 || name.length() > 190) throw bad("第" + line + "行编码或名称为空/过长"); if (code.isBlank() || name.isBlank() || code.length() > 100 || name.length() > 190) throw bad("第" + line + "行编码或名称为空/过长");
if (imported.containsKey(code) || !names.add(name)) throw bad("第" + line + "行编码或名称重复,未写入草稿"); if (imported.containsKey(code) || !names.add(name)) throw bad("第" + line + "行编码或名称重复,未写入草稿");
imported.put(code, new LinkedHashMap<>(Map.of("code", code, "name", name, "enabled", enabled(text(source, columns, "enabled"), line)))); Map<String, Object> item = new LinkedHashMap<>(existing.getOrDefault(code, Map.of()));
item.putAll(Map.of("code", code, "name", name, "enabled", enabled(text(source, columns, "enabled"), line)));
imported.put(code, item);
} }
int added = 0, updated = 0, disabled = 0; int added = 0, updated = 0, disabled = 0;
for (var row : imported.entrySet()) { if (existing.containsKey(row.getKey())) updated++; else added++; } for (var row : imported.entrySet()) { if (existing.containsKey(row.getKey())) updated++; else added++; }
@@ -48,7 +48,7 @@ public class MetricsController {
return calculate(year,month,quarter,department,group,product,productSearch,projectGroup,view,includeUnassigned,releaseId,trendGranularity,rowLimit,rowOffset,rowOrder,extraFilters,exclusions,null,actor); return calculate(year,month,quarter,department,group,product,productSearch,projectGroup,view,includeUnassigned,releaseId,trendGranularity,rowLimit,rowOffset,rowOrder,extraFilters,exclusions,null,actor);
} }
@GetMapping @GetMapping
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public MetricsService.MetricSnapshot calculateHttp(@RequestParam(required = false) Integer year, public MetricsService.MetricSnapshot calculateHttp(@RequestParam(required = false) Integer year,
@RequestParam(required = false) Integer month, @RequestParam(required = false) Integer month,
@RequestParam(required = false) Integer monthStart, @RequestParam(required = false) Integer monthStart,
@@ -70,15 +70,17 @@ public class MetricsController {
@RequestParam(required=false) String exclusions, @RequestParam(required=false) String exclusions,
@RequestParam(required=false) String columnFilters, @RequestParam(required=false) String columnFilters,
@RequestParam(required=false) String stage, @RequestParam(required=false) String stage,
@RequestParam(defaultValue="full") String part,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
if (!List.of("full", "table").contains(part)) throw new IllegalArgumentException("part 必须是 full 或 table");
return calculateRangeStage(year, month, monthStart, monthEnd, quarter, department, group, product, productSearch, return calculateRangeStage(year, month, monthStart, monthEnd, quarter, department, group, product, productSearch,
projectGroup, view, includeUnassigned, releaseId, trendGranularity, rowLimit, rowOffset, rowOrder, projectGroup, view, includeUnassigned, releaseId, trendGranularity, rowLimit, rowOffset, rowOrder,
extraFilters, exclusions, columnFilters, stage, actor); extraFilters, exclusions, columnFilters, stage, actor, "table".equals(part));
} }
/** 轻量趋势接口:查询参数与正式指标接口一致,但只返回版本、上下文、趋势和覆盖信息。 */ /** 轻量趋势接口:查询参数与正式指标接口一致,但只返回版本、上下文、趋势和覆盖信息。 */
@GetMapping("/trend") @GetMapping("/trend")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public MetricsService.MetricTrendSnapshot trendHttp(@RequestParam(required = false) Integer year, public MetricsService.MetricTrendSnapshot trendHttp(@RequestParam(required = false) Integer year,
@RequestParam(required = false) Integer month, @RequestParam(required = false) Integer month,
@RequestParam(required = false) Integer monthStart, @RequestParam(required = false) Integer monthStart,
@@ -131,10 +133,15 @@ public class MetricsController {
} }
private MetricsService.MetricSnapshot calculateRangeStage(Integer year,Integer month,Integer monthStart,Integer monthEnd,String quarter,String department,String group,String product,String productSearch,String projectGroup,String view,boolean includeUnassigned,String releaseId,String trendGranularity,Integer rowLimit,int rowOffset,String rowOrder,String extraFilters,String exclusions,String columnFilters,String stage,AppUserPrincipal actor) { private MetricsService.MetricSnapshot calculateRangeStage(Integer year,Integer month,Integer monthStart,Integer monthEnd,String quarter,String department,String group,String product,String productSearch,String projectGroup,String view,boolean includeUnassigned,String releaseId,String trendGranularity,Integer rowLimit,int rowOffset,String rowOrder,String extraFilters,String exclusions,String columnFilters,String stage,AppUserPrincipal actor) {
return calculateRangeStage(year,month,monthStart,monthEnd,quarter,department,group,product,productSearch,projectGroup,view,includeUnassigned,releaseId,trendGranularity,rowLimit,rowOffset,rowOrder,extraFilters,exclusions,columnFilters,stage,actor,false);
}
private MetricsService.MetricSnapshot calculateRangeStage(Integer year,Integer month,Integer monthStart,Integer monthEnd,String quarter,String department,String group,String product,String productSearch,String projectGroup,String view,boolean includeUnassigned,String releaseId,String trendGranularity,Integer rowLimit,int rowOffset,String rowOrder,String extraFilters,String exclusions,String columnFilters,String stage,AppUserPrincipal actor,boolean tableOnly) {
Integer quarterNumber = parseQuarter(quarter); Integer quarterNumber = parseQuarter(quarter);
MetricsService.MetricSnapshot visible = responseAccess.visible(metrics.calculate(new MetricsService.MetricsQuery(year, month, quarterNumber, department, group, MetricsService.MetricsQuery query = new MetricsService.MetricsQuery(year, month, quarterNumber, department, group,
product, projectGroup, view, includeUnassigned, releaseId, trendGranularity, parseExclusions(exclusions)) product, projectGroup, view, includeUnassigned, releaseId, trendGranularity, parseExclusions(exclusions))
.withMonthRange(monthStart, monthEnd).withExtraFilters(extraFilters).withProductSearch(productSearch).withColumnFilters(columnFilters).withStage(stage), actor), actor); .withMonthRange(monthStart, monthEnd).withExtraFilters(extraFilters).withProductSearch(productSearch).withColumnFilters(columnFilters).withStage(stage);
MetricsService.MetricSnapshot visible = responseAccess.visible(tableOnly ? metrics.calculateTable(query,actor) : metrics.calculate(query,actor), actor);
if (rowLimit == null) return visible; if (rowLimit == null) return visible;
if (rowLimit < 1 || rowLimit > 500 || rowOffset < 0) throw new IllegalArgumentException("rowLimit 必须在 1 到 500 之间,rowOffset 不能为负数"); if (rowLimit < 1 || rowLimit > 500 || rowOffset < 0) throw new IllegalArgumentException("rowLimit 必须在 1 到 500 之间,rowOffset 不能为负数");
if (rowOrder != null && !rowOrder.equals("scrapDesc")) throw new IllegalArgumentException("rowOrder 无效"); if (rowOrder != null && !rowOrder.equals("scrapDesc")) throw new IllegalArgumentException("rowOrder 无效");
@@ -1,13 +1,21 @@
package cn.yunsheng.quality.metrics; package cn.yunsheng.quality.metrics;
import cn.yunsheng.quality.filters.SourceFilterIndex;
import java.util.List; import java.util.List;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.jdbc.core.JdbcTemplate; import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional; import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.support.TransactionSynchronization;
import org.springframework.transaction.support.TransactionSynchronizationManager;
/** Builds immutable, per-transform-build source-component rollups for read-only metric queries. */ /** Builds immutable, per-transform-build source-component rollups for read-only metric queries. */
@Service @Service
public class MetricsRollupService { public class MetricsRollupService {
private static final Logger LOG = LoggerFactory.getLogger(MetricsRollupService.class);
static final String DIMENSION_COLLATION = "utf8mb4_0900_as_cs"; static final String DIMENSION_COLLATION = "utf8mb4_0900_as_cs";
private static final List<String> FACT_TABLES = List.of("fact_black", "fact_bright"); private static final List<String> FACT_TABLES = List.of("fact_black", "fact_bright");
private static final String COMPONENTS = "JSON_OBJECT(" + private static final String COMPONENTS = "JSON_OBJECT(" +
@@ -22,21 +30,32 @@ public class MetricsRollupService {
private final JdbcTemplate jdbc; private final JdbcTemplate jdbc;
private final cn.yunsheng.quality.analysis.FineDefectRollups fineDefects; private final cn.yunsheng.quality.analysis.FineDefectRollups fineDefects;
private final SourceFilterIndex sourceFilterIndex;
private ApplicationEventPublisher eventPublisher;
public MetricsRollupService(JdbcTemplate jdbc) { public MetricsRollupService(JdbcTemplate jdbc) {
this.jdbc = jdbc; this.jdbc = jdbc;
this.fineDefects = new cn.yunsheng.quality.analysis.FineDefectRollups(jdbc, tools.jackson.databind.json.JsonMapper.builder().build()); this.fineDefects = new cn.yunsheng.quality.analysis.FineDefectRollups(jdbc, tools.jackson.databind.json.JsonMapper.builder().build());
this.sourceFilterIndex = new SourceFilterIndex(jdbc);
}
@Autowired
public void setApplicationEventPublisher(ApplicationEventPublisher eventPublisher) {
this.eventPublisher = eventPublisher;
} }
public record GroupKey(String stage,String month,String department,String group,String project,String product) {} public record GroupKey(String stage,String month,String department,String group,String project,String product) {}
/** 归类维护沿用旧汇总;仅重建归属发生变化的分组,重量计算结果不变。 */ /** 归类维护沿用旧汇总;仅重建归属发生变化的分组,重量计算结果不变。 */
@Transactional
public void copyBuild(String parent,String child) { public void copyBuild(String parent,String child) {
jdbc.update("INSERT INTO metric_rollup(build_id,stage,month_key,department,group_name,project_group,product,qualified_weight,scrap_weight,total_weight,components,source_row_count,excluded_blank_rows,unmapped_rows,project_unconfigured_rows) SELECT ?,stage,month_key,department,group_name,project_group,product,qualified_weight,scrap_weight,total_weight,components,source_row_count,excluded_blank_rows,unmapped_rows,project_unconfigured_rows FROM metric_rollup WHERE build_id=?",child,parent); jdbc.update("INSERT INTO metric_rollup(build_id,stage,month_key,department,group_name,project_group,product,qualified_weight,scrap_weight,total_weight,components,source_row_count,excluded_blank_rows,unmapped_rows,project_unconfigured_rows) SELECT ?,stage,month_key,department,group_name,project_group,product,qualified_weight,scrap_weight,total_weight,components,source_row_count,excluded_blank_rows,unmapped_rows,project_unconfigured_rows FROM metric_rollup WHERE build_id=?",child,parent);
if(jdbc.update("INSERT INTO metric_rollup_build(build_id,source_row_count) SELECT ?,source_row_count FROM metric_rollup_build WHERE build_id=?",child,parent)!=1) throw new IllegalStateException("旧版本缺少已冻结的指标汇总"); if(jdbc.update("INSERT INTO metric_rollup_build(build_id,source_row_count) SELECT ?,source_row_count FROM metric_rollup_build WHERE build_id=?",child,parent)!=1) throw new IllegalStateException("旧版本缺少已冻结的指标汇总");
fineDefects.copyBuild(parent, child); fineDefects.copyBuild(parent, child);
announceSourceFilterBuild(child);
} }
@Transactional
public void refreshGroups(String build,java.util.Set<GroupKey> keys) { public void refreshGroups(String build,java.util.Set<GroupKey> keys) {
for(String stage:List.of("black","bright")) { for(String stage:List.of("black","bright")) {
var selected=keys.stream().filter(k->stage.equals(k.stage())).toList();if(selected.isEmpty())continue; var selected=keys.stream().filter(k->stage.equals(k.stage())).toList();if(selected.isEmpty())continue;
@@ -49,6 +68,7 @@ public class MetricsRollupService {
jdbc.update(rollupInsertSql("fact_"+stage).replace("WHERE f.build_id=?","WHERE f.build_id=? AND ("+inserted+")"),insertArgs.toArray()); jdbc.update(rollupInsertSql("fact_"+stage).replace("WHERE f.build_id=?","WHERE f.build_id=? AND ("+inserted+")"),insertArgs.toArray());
} }
fineDefects.refreshGroups(build, keys); fineDefects.refreshGroups(build, keys);
new SalesOrderMetricRollups(jdbc).refreshCurrentBuild(build);
} }
/** Idempotently refreshes a completed build in the caller's transaction. */ /** Idempotently refreshes a completed build in the caller's transaction. */
@@ -70,6 +90,31 @@ public class MetricsRollupService {
(SELECT COUNT(*) FROM fact_bright WHERE build_id=?), CURRENT_TIMESTAMP(6) (SELECT COUNT(*) FROM fact_bright WHERE build_id=?), CURRENT_TIMESTAMP(6)
""", buildId, buildId, buildId); """, buildId, buildId, buildId);
fineDefects.refreshBuild(buildId); fineDefects.refreshBuild(buildId);
new SalesOrderMetricRollups(jdbc).refreshCurrentBuild(buildId);
announceSourceFilterBuild(buildId);
}
void announceSourceFilterBuild(String buildId) {
if (eventPublisher == null) {
// Directly constructed fixtures have no application event bus; retain
// their explicit synchronous behavior (legacy schemas no-op safely).
sourceFilterIndex.ensureBuild(buildId);
return;
}
Runnable publish = () -> {
try {
eventPublisher.publishEvent(new SourceFilterIndex.BuildAvailable(buildId));
} catch (RuntimeException failure) {
LOG.error("Source filter index notification failed for {}; raw-value fallback remains available", buildId, failure);
}
};
if (TransactionSynchronizationManager.isSynchronizationActive()) {
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
@Override public void afterCommit() { publish.run(); }
});
} else {
publish.run();
}
} }
static String rollupInsertSql(String table) { static String rollupInsertSql(String table) {
@@ -39,6 +39,10 @@ public class MetricsService {
private static final MathContext RATE_CONTEXT = MathContext.DECIMAL128; private static final MathContext RATE_CONTEXT = MathContext.DECIMAL128;
private final JdbcTemplate jdbc; private final JdbcTemplate jdbc;
private final GovernanceAccessService governance; private final GovernanceAccessService governance;
private SourceColumnQueries sourceColumns;
@Autowired
public void setSourceColumns(SourceColumnQueries sourceColumns) { this.sourceColumns = sourceColumns; }
private final TransactionTemplate readSnapshot; private final TransactionTemplate readSnapshot;
private final TransactionTemplate governanceSnapshot; private final TransactionTemplate governanceSnapshot;
@@ -46,6 +50,7 @@ public class MetricsService {
public MetricsService(JdbcTemplate jdbc, GovernanceAccessService governance, PlatformTransactionManager transactionManager) { public MetricsService(JdbcTemplate jdbc, GovernanceAccessService governance, PlatformTransactionManager transactionManager) {
this.jdbc = jdbc; this.jdbc = jdbc;
this.governance = governance; this.governance = governance;
this.sourceColumns = new SourceColumnQueries(jdbc,governance,JsonMapper.builder().build());
this.readSnapshot = new TransactionTemplate(transactionManager); this.readSnapshot = new TransactionTemplate(transactionManager);
this.readSnapshot.setIsolationLevel(TransactionDefinition.ISOLATION_REPEATABLE_READ); this.readSnapshot.setIsolationLevel(TransactionDefinition.ISOLATION_REPEATABLE_READ);
this.readSnapshot.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRES_NEW); this.readSnapshot.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRES_NEW);
@@ -56,10 +61,21 @@ public class MetricsService {
/** Keeps package-level in-memory aggregation tests independent of a database transaction manager. */ /** Keeps package-level in-memory aggregation tests independent of a database transaction manager. */
public MetricsService(JdbcTemplate jdbc, GovernanceAccessService governance) { public MetricsService(JdbcTemplate jdbc, GovernanceAccessService governance) {
this.jdbc = jdbc; this.governance = governance; this.readSnapshot = null; this.governanceSnapshot = null; this.jdbc = jdbc; this.governance = governance;
this.sourceColumns = new SourceColumnQueries(jdbc,governance,JsonMapper.builder().build());
this.readSnapshot = null; this.governanceSnapshot = null;
} }
public MetricSnapshot calculate(MetricsQuery query, AppUserPrincipal actor) { public MetricSnapshot calculate(MetricsQuery query, AppUserPrincipal actor) {
return calculate(query, actor, false);
}
/** 仅按对象汇总表格,省去时间与对象交叉分组的中间行。 */
public MetricSnapshot calculateTable(MetricsQuery query, AppUserPrincipal actor) {
return calculate(query, actor, true);
}
private MetricSnapshot calculate(MetricsQuery query, AppUserPrincipal actor, boolean tableOnly) {
if (actor == null) throw new ResponseStatusException(HttpStatus.FORBIDDEN, "指标查询必须绑定当前有效用户"); if (actor == null) throw new ResponseStatusException(HttpStatus.FORBIDDEN, "指标查询必须绑定当前有效用户");
governance.requireUserActivePolicy(actor); governance.requireUserActivePolicy(actor);
governance.requireBoundAction("QUERY", "query.aggregate"); governance.requireBoundAction("QUERY", "query.aggregate");
@@ -68,7 +84,7 @@ public class MetricsService {
? currentReleaseId() : query.releaseId(); ? currentReleaseId() : query.releaseId();
if (releaseId == null) return emptySnapshot(query); if (releaseId == null) return emptySnapshot(query);
ReleaseInfo release = release(releaseId, actor); ReleaseInfo release = release(releaseId, actor);
return calculateForRelease(query, actor, release); return withConsistentSnapshot(() -> calculateForReleaseSnapshot(query, actor, release, tableOnly));
} }
/** 只按时间桶生成趋势,不构造总体摘要或产品/分组维度行。 */ /** 只按时间桶生成趋势,不构造总体摘要或产品/分组维度行。 */
@@ -84,6 +100,23 @@ public class MetricsService {
return withConsistentSnapshot(() -> calculateTrendForReleaseSnapshot(query, actor, release)); return withConsistentSnapshot(() -> calculateTrendForReleaseSnapshot(query, actor, release));
} }
public record PairedSnapshot<T>(MetricSnapshot metrics, MetricTrendSnapshot trend, T source) {}
/** Authorize before opening the read snapshot, so service-binding locks do not span source scans. */
public <T> PairedSnapshot<T> calculatePairedSnapshot(MetricsQuery query, AppUserPrincipal actor,
boolean trendOnly, Supplier<T> source) {
if (actor == null) throw new ResponseStatusException(HttpStatus.FORBIDDEN,"指标查询必须绑定当前有效用户");
governance.requireUserActivePolicy(actor);
governance.requireBoundAction("QUERY","query.aggregate");
validate(query);
String releaseId=query.releaseId()==null||query.releaseId().isBlank()?currentReleaseId():query.releaseId();
if(releaseId==null)throw new ResponseStatusException(HttpStatus.BAD_REQUEST,"必须绑定正式数据版本");
ReleaseInfo release=release(releaseId,actor);
return withConsistentSnapshot(() -> trendOnly
? new PairedSnapshot<T>(null,calculateTrendForReleaseSnapshot(query,actor,release),source.get())
: new PairedSnapshot<T>(calculateForReleaseSnapshot(query,actor,release),null,source.get()));
}
private MetricTrendSnapshot calculateTrendForReleaseSnapshot(MetricsQuery query, AppUserPrincipal actor, private MetricTrendSnapshot calculateTrendForReleaseSnapshot(MetricsQuery query, AppUserPrincipal actor,
ReleaseInfo release) { ReleaseInfo release) {
boolean hasUnrolledBuilds = hasUnrolledBuilds(release.id()); boolean hasUnrolledBuilds = hasUnrolledBuilds(release.id());
@@ -111,21 +144,22 @@ public class MetricsService {
AppUserPrincipal actor, boolean hasUnrolledBuilds) { AppUserPrincipal actor, boolean hasUnrolledBuilds) {
Map<String, AggregateStats> periods = new TreeMap<>(); Map<String, AggregateStats> periods = new TreeMap<>();
boolean hasSourceFilters = !query.extraFilters().isEmpty() || !query.columnFilters().isEmpty(); boolean hasSourceFilters = !query.extraFilters().isEmpty() || !query.columnFilters().isEmpty();
if (!hasSourceFilters) { boolean salesRollup = salesOrderRollupReady(releaseId, query, actor);
aggregateTrendStage(periods, "black", releaseId, query, actor, false); if (!hasSourceFilters || salesRollup) {
aggregateTrendStage(periods, "bright", releaseId, query, actor, false); aggregateTrendStage(periods, "black", releaseId, query, actor, false, salesRollup);
aggregateTrendStage(periods, "bright", releaseId, query, actor, false, salesRollup);
} }
if (hasUnrolledBuilds || hasSourceFilters) { if (!salesRollup && (hasUnrolledBuilds || hasSourceFilters)) {
aggregateTrendStage(periods, "black", releaseId, query, actor, true); aggregateTrendStage(periods, "black", releaseId, query, actor, true, false);
aggregateTrendStage(periods, "bright", releaseId, query, actor, true); aggregateTrendStage(periods, "bright", releaseId, query, actor, true, false);
} }
return periods; return periods;
} }
private void aggregateTrendStage(Map<String, AggregateStats> periods, String stage, String releaseId, private void aggregateTrendStage(Map<String, AggregateStats> periods, String stage, String releaseId,
MetricsQuery query, AppUserPrincipal actor, boolean factFallback) { MetricsQuery query, AppUserPrincipal actor, boolean factFallback, boolean salesRollup) {
if (query.stage() != null && !query.stage().equals(stage)) return; if (query.stage() != null && !query.stage().equals(stage)) return;
String table = factFallback ? (stage.equals("black") ? "fact_black" : "fact_bright") : "metric_rollup"; String table = factFallback ? (stage.equals("black") ? "fact_black" : "fact_bright") : (salesRollup ? "sales_order_metric_rollup" : "metric_rollup");
StringBuilder where = new StringBuilder(" WHERE rm.release_id=?"); StringBuilder where = new StringBuilder(" WHERE rm.release_id=?");
List<Object> args = new ArrayList<>(); List<Object> args = new ArrayList<>();
args.add(releaseId); args.add(releaseId);
@@ -133,13 +167,14 @@ public class MetricsService {
if (factFallback && !hasSourceFilters) if (factFallback && !hasSourceFilters)
where.append(" AND NOT EXISTS (SELECT 1 FROM metric_rollup_build rb WHERE rb.build_id=f.build_id)"); where.append(" AND NOT EXISTS (SELECT 1 FROM metric_rollup_build rb WHERE rb.build_id=f.build_id)");
else if (!factFallback) { where.append(" AND f.stage=?"); args.add(stage); } else if (!factFallback) { where.append(" AND f.stage=?"); args.add(stage); }
if (salesRollup) new SalesOrderMetricRollups(jdbc).append(where, args, query.extraFilters());
appendTime(where, args, query); appendTime(where, args, query);
appendDimensions(where, args, query); appendDimensions(where, args, query);
appendExclusions(where, args, query); appendExclusions(where, args, query);
if (factFallback) { if (factFallback) {
new QueryFilterFields(jdbc, governance).append(where, args, stage, query.extraFilters(), actor); new QueryFilterFields(jdbc, governance).append(where, args, stage, query.extraFilters(), actor,releaseId);
if (!query.columnFilters().isEmpty()) if (!query.columnFilters().isEmpty())
new SourceColumnQueries(jdbc, governance, JsonMapper.builder().build()) sourceColumns
.append(where, args, stage, query.columnFilters(), releaseId, actor); .append(where, args, stage, query.columnFilters(), releaseId, actor);
} }
DataScope.append(where, args, actor, "f.department COLLATE utf8mb4_0900_as_cs", DataScope.append(where, args, actor, "f.department COLLATE utf8mb4_0900_as_cs",
@@ -161,7 +196,7 @@ public class MetricsService {
%s AS bright_processing_scrap, %s AS bright_processing_scrap,
%s AS bright_diffusion_scrap, %s AS bright_diffusion_scrap,
%s AS row_count %s AS row_count
FROM release_member rm JOIN %s f ON f.build_id=rm.transform_build_id FROM release_member rm JOIN %s f%s ON f.build_id=rm.transform_build_id
%s %s
GROUP BY trend_period GROUP BY trend_period
ORDER BY trend_period ORDER BY trend_period
@@ -170,7 +205,7 @@ public class MetricsService {
componentSum("black_external_scrap"), componentSum("bright_headquarters_scrap"), componentSum("black_external_scrap"), componentSum("bright_headquarters_scrap"),
componentSum("bright_plating_scrap"), componentSum("bright_material_scrap"), componentSum("bright_plating_scrap"), componentSum("bright_material_scrap"),
componentSum("bright_processing_scrap"), componentSum("bright_diffusion_scrap"), componentSum("bright_processing_scrap"), componentSum("bright_diffusion_scrap"),
countExpression, table, where); countExpression, table, factFilterHint(stage,factFallback,query), where);
jdbc.query(sql, (RowCallbackHandler) rs -> { jdbc.query(sql, (RowCallbackHandler) rs -> {
String period = rs.getString("trend_period"); String period = rs.getString("trend_period");
AggregateStats stats = new AggregateStats().read(rs, stage); AggregateStats stats = new AggregateStats().read(rs, stage);
@@ -192,7 +227,7 @@ public class MetricsService {
if (actor == null) throw new ResponseStatusException(HttpStatus.FORBIDDEN, "对账指标查询必须绑定当前有效用户"); if (actor == null) throw new ResponseStatusException(HttpStatus.FORBIDDEN, "对账指标查询必须绑定当前有效用户");
governance.requireUserActivePolicy(actor); governance.requireUserActivePolicy(actor);
governance.requireBoundAction("QUERY", "query.aggregate"); governance.requireBoundAction("QUERY", "query.aggregate");
Set<String> allowedRoles = Set.of("DATA_MAINTAINER", "RULE_MAINTAINER", "DATA_PUBLISHER", "SYSTEM_ADMIN"); Set<String> allowedRoles = Set.of("DATA_ADMIN", "SYSTEM_ADMIN");
if (actor.getRoleCodes().stream().noneMatch(allowedRoles::contains)) if (actor.getRoleCodes().stream().noneMatch(allowedRoles::contains))
throw new ResponseStatusException(HttpStatus.FORBIDDEN, "当前功能角色不允许执行候选基线对账"); throw new ResponseStatusException(HttpStatus.FORBIDDEN, "当前功能角色不允许执行候选基线对账");
validate(query); validate(query);
@@ -218,9 +253,13 @@ public class MetricsService {
} }
private MetricSnapshot calculateForReleaseSnapshot(MetricsQuery query, AppUserPrincipal actor, ReleaseInfo release) { private MetricSnapshot calculateForReleaseSnapshot(MetricsQuery query, AppUserPrincipal actor, ReleaseInfo release) {
return calculateForReleaseSnapshot(query, actor, release, false);
}
private MetricSnapshot calculateForReleaseSnapshot(MetricsQuery query, AppUserPrincipal actor, ReleaseInfo release, boolean tableOnly) {
String releaseId = release.id(); String releaseId = release.id();
boolean hasUnrolledBuilds = hasUnrolledBuilds(releaseId); boolean hasUnrolledBuilds = hasUnrolledBuilds(releaseId);
AggregateSet aggregates = queryAggregates(releaseId, query, actor, hasUnrolledBuilds); AggregateSet aggregates = queryAggregates(releaseId, query, actor, hasUnrolledBuilds, null, !tableOnly);
String ruleVersion=jdbc.queryForObject("SELECT rule_version FROM data_release WHERE id=?",String.class,releaseId); String ruleVersion=jdbc.queryForObject("SELECT rule_version FROM data_release WHERE id=?",String.class,releaseId);
ConfigurableRuleEngine rules=rulesForVersion(ruleVersion); ConfigurableRuleEngine rules=rulesForVersion(ruleVersion);
MetricRow summary = rules.applyMetrics(aggregates.summary().toMetricRow("overall", "总体", null, null, null)); MetricRow summary = rules.applyMetrics(aggregates.summary().toMetricRow("overall", "总体", null, null, null));
@@ -311,6 +350,12 @@ public class MetricsService {
return rawRows - (effectiveIncludeUnassigned(query) ? 0 : excludedBlankRows); return rawRows - (effectiveIncludeUnassigned(query) ? 0 : excludedBlankRows);
} }
private boolean salesOrderRollupReady(String release, MetricsQuery query, AppUserPrincipal actor) {
if (query.extraFilters().isEmpty() || !query.columnFilters().isEmpty()
|| !Set.of(cn.yunsheng.quality.filters.SalesOrderTypes.KEY, cn.yunsheng.quality.filters.SalesOrderTypes.EXCLUDE_KEY).containsAll(query.extraFilters().keySet())) return false;
return new SalesOrderMetricRollups(jdbc).ready(release, query.extraFilters(), false, governance.allowedFields(actor));
}
private boolean hasUnrolledBuilds(String releaseId) { private boolean hasUnrolledBuilds(String releaseId) {
Integer count = jdbc.queryForObject(""" Integer count = jdbc.queryForObject("""
SELECT COUNT(*) FROM release_member rm SELECT COUNT(*) FROM release_member rm
@@ -326,21 +371,30 @@ public class MetricsService {
private AggregateSet queryAggregates(String releaseId, MetricsQuery query, AppUserPrincipal actor, private AggregateSet queryAggregates(String releaseId, MetricsQuery query, AppUserPrincipal actor,
boolean hasUnrolledBuilds, Map<String, String> canonicalGroupDepartments) { boolean hasUnrolledBuilds, Map<String, String> canonicalGroupDepartments) {
return queryAggregates(releaseId, query, actor, hasUnrolledBuilds, canonicalGroupDepartments, true);
}
private AggregateSet queryAggregates(String releaseId, MetricsQuery query, AppUserPrincipal actor,
boolean hasUnrolledBuilds, Map<String, String> canonicalGroupDepartments, boolean includePeriods) {
AggregateSet result = new AggregateSet(); AggregateSet result = new AggregateSet();
if ((query.extraFilters().isEmpty() && query.columnFilters().isEmpty())) aggregateStage(result, "black", query.view(), releaseId, query, actor, false, canonicalGroupDepartments); boolean salesRollup = salesOrderRollupReady(releaseId, query, actor);
if ((query.extraFilters().isEmpty() && query.columnFilters().isEmpty())) aggregateStage(result, "bright", query.view(), releaseId, query, actor, false, canonicalGroupDepartments); boolean hasSourceFilters = !query.extraFilters().isEmpty() || !query.columnFilters().isEmpty();
if (hasUnrolledBuilds || !(query.extraFilters().isEmpty() && query.columnFilters().isEmpty())) { if (!hasSourceFilters || salesRollup) {
aggregateStage(result, "black", query.view(), releaseId, query, actor, true, canonicalGroupDepartments); aggregateStage(result, "black", query.view(), releaseId, query, actor, false, salesRollup, canonicalGroupDepartments, includePeriods);
aggregateStage(result, "bright", query.view(), releaseId, query, actor, true, canonicalGroupDepartments); aggregateStage(result, "bright", query.view(), releaseId, query, actor, false, salesRollup, canonicalGroupDepartments, includePeriods);
}
if (!salesRollup && (hasUnrolledBuilds || hasSourceFilters)) {
aggregateStage(result, "black", query.view(), releaseId, query, actor, true, false, canonicalGroupDepartments, includePeriods);
aggregateStage(result, "bright", query.view(), releaseId, query, actor, true, false, canonicalGroupDepartments, includePeriods);
} }
return result; return result;
} }
private void aggregateStage(AggregateSet result, String stage, String view, String releaseId, private void aggregateStage(AggregateSet result, String stage, String view, String releaseId,
MetricsQuery query, AppUserPrincipal actor, boolean factFallback, MetricsQuery query, AppUserPrincipal actor, boolean factFallback, boolean salesRollup,
Map<String, String> canonicalGroupDepartments) { Map<String, String> canonicalGroupDepartments, boolean includePeriods) {
if(query.stage()!=null&&!query.stage().equals(stage))return; if(query.stage()!=null&&!query.stage().equals(stage))return;
String table = factFallback ? (stage.equals("black") ? "fact_black" : "fact_bright") : "metric_rollup"; String table = factFallback ? (stage.equals("black") ? "fact_black" : "fact_bright") : (salesRollup ? "sales_order_metric_rollup" : "metric_rollup");
String dimensionColumn = view.equals("overall") ? null : switch (view) { String dimensionColumn = view.equals("overall") ? null : switch (view) {
case "department" -> "f.department"; case "department" -> "f.department";
case "group" -> "f.group_name"; case "group" -> "f.group_name";
@@ -358,17 +412,18 @@ public class MetricsService {
args.add(releaseId); args.add(releaseId);
if (factFallback && (query.extraFilters().isEmpty() && query.columnFilters().isEmpty())) where.append(" AND NOT EXISTS (SELECT 1 FROM metric_rollup_build rb WHERE rb.build_id=f.build_id)"); if (factFallback && (query.extraFilters().isEmpty() && query.columnFilters().isEmpty())) where.append(" AND NOT EXISTS (SELECT 1 FROM metric_rollup_build rb WHERE rb.build_id=f.build_id)");
else if (!factFallback) { where.append(" AND f.stage=?"); args.add(stage); } else if (!factFallback) { where.append(" AND f.stage=?"); args.add(stage); }
if (salesRollup) new SalesOrderMetricRollups(jdbc).append(where, args, query.extraFilters());
appendTime(where, args, query); appendTime(where, args, query);
appendDimensions(where, args, query); appendDimensions(where, args, query);
appendExclusions(where, args, query); appendExclusions(where, args, query);
if(factFallback) { if(factFallback) {
new QueryFilterFields(jdbc,governance).append(where,args,stage,query.extraFilters(),actor); new QueryFilterFields(jdbc,governance).append(where,args,stage,query.extraFilters(),actor,releaseId);
if(!query.columnFilters().isEmpty())new SourceColumnQueries(jdbc,governance,JsonMapper.builder().build()).append(where,args,stage,query.columnFilters(),releaseId,actor); if(!query.columnFilters().isEmpty())sourceColumns.append(where,args,stage,query.columnFilters(),releaseId,actor);
} }
DataScope.append(where, args, actor, "f.department COLLATE utf8mb4_0900_as_cs", "f.group_name COLLATE utf8mb4_0900_as_cs"); DataScope.append(where, args, actor, "f.department COLLATE utf8mb4_0900_as_cs", "f.group_name COLLATE utf8mb4_0900_as_cs");
if (!effectiveIncludeUnassigned(query)) if (!effectiveIncludeUnassigned(query))
where.append(" AND (f.group_name IS NOT NULL AND f.group_name<>'')"); where.append(" AND (f.group_name IS NOT NULL AND f.group_name<>'')");
String trendBucket = trendBucketExpression(query.trendGranularity()); String trendBucket = includePeriods ? trendBucketExpression(query.trendGranularity()) : "'all'";
String sql = """ String sql = """
SELECT %s AS trend_period, %s AS bucket_key%s, SELECT %s AS trend_period, %s AS bucket_key%s,
SUM(f.qualified_weight) AS qualified_weight, SUM(f.qualified_weight) AS qualified_weight,
@@ -383,7 +438,7 @@ public class MetricsService {
%s AS bright_processing_scrap, %s AS bright_processing_scrap,
%s AS bright_diffusion_scrap, %s AS bright_diffusion_scrap,
%s AS row_count %s AS row_count
FROM release_member rm JOIN %s f ON f.build_id=rm.transform_build_id FROM release_member rm JOIN %s f%s ON f.build_id=rm.transform_build_id
%s %s
GROUP BY %s GROUP BY %s
ORDER BY %s ORDER BY %s
@@ -393,14 +448,14 @@ public class MetricsService {
componentSum("black_external_scrap"), componentSum("bright_headquarters_scrap"), componentSum("black_external_scrap"), componentSum("bright_headquarters_scrap"),
componentSum("bright_plating_scrap"), componentSum("bright_material_scrap"), componentSum("bright_plating_scrap"), componentSum("bright_material_scrap"),
componentSum("bright_processing_scrap"), componentSum("bright_diffusion_scrap"), componentSum("bright_processing_scrap"), componentSum("bright_diffusion_scrap"),
factFallback ? "COUNT(*)" : "SUM(f.source_row_count)", table, where, factFallback ? "COUNT(*)" : "SUM(f.source_row_count)", table, factFilterHint(stage,factFallback,query), where,
aggregateKeys, aggregateKeys); aggregateKeys, aggregateKeys);
jdbc.query(sql, (RowCallbackHandler) rs -> { jdbc.query(sql, (RowCallbackHandler) rs -> {
String period = rs.getString("trend_period"); String period = rs.getString("trend_period");
String key = rs.getString("bucket_key"); String key = rs.getString("bucket_key");
AggregateStats stats = new AggregateStats().read(rs, stage); AggregateStats stats = new AggregateStats().read(rs, stage);
result.summary().merge(stage, stats); result.summary().merge(stage, stats);
result.periods().computeIfAbsent(period, ignored -> new AggregateStats()).merge(stage, stats); if (includePeriods) result.periods().computeIfAbsent(period, ignored -> new AggregateStats()).merge(stage, stats);
if (dimensionColumn != null) { if (dimensionColumn != null) {
if (!canonicalGroupRows || canonicalGroupOwnerMatches(canonicalGroupDepartments, key, rs.getString("owner_department"))) if (!canonicalGroupRows || canonicalGroupOwnerMatches(canonicalGroupDepartments, key, rs.getString("owner_department")))
result.dimensions().computeIfAbsent(key, ignored -> new AggregateStats()).merge(stage, stats); result.dimensions().computeIfAbsent(key, ignored -> new AggregateStats()).merge(stage, stats);
@@ -521,11 +576,15 @@ public class MetricsService {
Map<String, Object> queryCoverage(String releaseId, MetricsQuery query, AppUserPrincipal actor, boolean hasUnrolledBuilds) { Map<String, Object> queryCoverage(String releaseId, MetricsQuery query, AppUserPrincipal actor, boolean hasUnrolledBuilds) {
TreeMap<String, CoverageMonth> months = new TreeMap<>(); TreeMap<String, CoverageMonth> months = new TreeMap<>();
if ((query.extraFilters().isEmpty() && query.columnFilters().isEmpty())) coverageStage(months, "black", releaseId, query, actor, false); boolean salesRollup = salesOrderRollupReady(releaseId, query, actor);
if ((query.extraFilters().isEmpty() && query.columnFilters().isEmpty())) coverageStage(months, "bright", releaseId, query, actor, false); boolean hasSourceFilters = !query.extraFilters().isEmpty() || !query.columnFilters().isEmpty();
if (hasUnrolledBuilds || !(query.extraFilters().isEmpty() && query.columnFilters().isEmpty())) { if (!hasSourceFilters || salesRollup) {
coverageStage(months, "black", releaseId, query, actor, true); coverageStage(months, "black", releaseId, query, actor, false, salesRollup);
coverageStage(months, "bright", releaseId, query, actor, true); coverageStage(months, "bright", releaseId, query, actor, false, salesRollup);
}
if (!salesRollup && (hasUnrolledBuilds || hasSourceFilters)) {
coverageStage(months, "black", releaseId, query, actor, true, false);
coverageStage(months, "bright", releaseId, query, actor, true, false);
} }
long rawRows = 0, excludedBlank = 0, unmapped = 0, projectUnconfigured = 0; long rawRows = 0, excludedBlank = 0, unmapped = 0, projectUnconfigured = 0;
List<Map<String, Object>> sourceMonths = new ArrayList<>(); List<Map<String, Object>> sourceMonths = new ArrayList<>();
@@ -561,21 +620,32 @@ public class MetricsService {
"sourceMonths", sourceMonths); "sourceMonths", sourceMonths);
} }
private String factFilterHint(String stage,boolean factFallback,MetricsQuery query) {
if(sourceColumns.indexed(query.releaseId()))return "";
if(!factFallback||query.columnFilters().isEmpty()||!query.extraFilters().isEmpty())return "";
// 仅纠正无组织/产品限制的整年来源筛选,窄范围和销售订单组合保留优化器原选择。
if(query.month()!=null||query.quarter()!=null||query.monthStart()!=null||query.monthEnd()!=null
||query.department()!=null||query.group()!=null||query.product()!=null||query.projectGroup()!=null
||query.productSearch()!=null)return "";
return SourceColumnQueries.factIndexHint(stage);
}
private void coverageStage(Map<String, CoverageMonth> months, String stage, String releaseId, private void coverageStage(Map<String, CoverageMonth> months, String stage, String releaseId,
MetricsQuery query, AppUserPrincipal actor, boolean factFallback) { MetricsQuery query, AppUserPrincipal actor, boolean factFallback, boolean salesRollup) {
if(query.stage()!=null&&!query.stage().equals(stage))return; if(query.stage()!=null&&!query.stage().equals(stage))return;
String table = factFallback ? (stage.equals("black") ? "fact_black" : "fact_bright") : "metric_rollup"; String table = factFallback ? (stage.equals("black") ? "fact_black" : "fact_bright") : (salesRollup ? "sales_order_metric_rollup" : "metric_rollup");
StringBuilder where = new StringBuilder(" WHERE rm.release_id=?"); StringBuilder where = new StringBuilder(" WHERE rm.release_id=?");
List<Object> args = new ArrayList<>(); List<Object> args = new ArrayList<>();
args.add(releaseId); args.add(releaseId);
if (factFallback && (query.extraFilters().isEmpty() && query.columnFilters().isEmpty())) where.append(" AND NOT EXISTS (SELECT 1 FROM metric_rollup_build rb WHERE rb.build_id=f.build_id)"); if (factFallback && (query.extraFilters().isEmpty() && query.columnFilters().isEmpty())) where.append(" AND NOT EXISTS (SELECT 1 FROM metric_rollup_build rb WHERE rb.build_id=f.build_id)");
else if (!factFallback) { where.append(" AND f.stage=?"); args.add(stage); } else if (!factFallback) { where.append(" AND f.stage=?"); args.add(stage); }
if (salesRollup) new SalesOrderMetricRollups(jdbc).append(where, args, query.extraFilters());
appendTime(where, args, query); appendTime(where, args, query);
appendDimensions(where, args, query); appendDimensions(where, args, query);
appendExclusions(where, args, query); appendExclusions(where, args, query);
if(factFallback) { if(factFallback) {
new QueryFilterFields(jdbc,governance).append(where,args,stage,query.extraFilters(),actor); new QueryFilterFields(jdbc,governance).append(where,args,stage,query.extraFilters(),actor,releaseId);
if(!query.columnFilters().isEmpty())new SourceColumnQueries(jdbc,governance,JsonMapper.builder().build()).append(where,args,stage,query.columnFilters(),releaseId,actor); if(!query.columnFilters().isEmpty())sourceColumns.append(where,args,stage,query.columnFilters(),releaseId,actor);
} }
DataScope.append(where, args, actor, "f.department COLLATE utf8mb4_0900_as_cs", "f.group_name COLLATE utf8mb4_0900_as_cs"); DataScope.append(where, args, actor, "f.department COLLATE utf8mb4_0900_as_cs", "f.group_name COLLATE utf8mb4_0900_as_cs");
String countExpression = factFallback ? "COUNT(*)" : "SUM(f.source_row_count)"; String countExpression = factFallback ? "COUNT(*)" : "SUM(f.source_row_count)";
@@ -590,11 +660,11 @@ public class MetricsService {
String sql = """ String sql = """
SELECT f.month_key, %s AS row_count, SELECT f.month_key, %s AS row_count,
%s AS excluded_blank_rows, %s AS unmapped_rows, %s AS project_unconfigured_rows %s AS excluded_blank_rows, %s AS unmapped_rows, %s AS project_unconfigured_rows
FROM release_member rm JOIN %s f ON f.build_id=rm.transform_build_id FROM release_member rm JOIN %s f%s ON f.build_id=rm.transform_build_id
%s %s
GROUP BY f.month_key GROUP BY f.month_key
ORDER BY f.month_key ORDER BY f.month_key
""".formatted(countExpression, excludedExpression, unmappedExpression, projectExpression, table, where); """.formatted(countExpression, excludedExpression, unmappedExpression, projectExpression, table, factFilterHint(stage,factFallback,query), where);
jdbc.query(sql, (RowCallbackHandler) rs -> { jdbc.query(sql, (RowCallbackHandler) rs -> {
CoverageMonth coverage = months.computeIfAbsent(rs.getString("month_key"), ignored -> new CoverageMonth()); CoverageMonth coverage = months.computeIfAbsent(rs.getString("month_key"), ignored -> new CoverageMonth());
long rowCount = rs.getLong("row_count"); long rowCount = rs.getLong("row_count");
@@ -765,6 +835,7 @@ public class MetricsService {
result.put("trendGranularity", query.trendGranularity()); result.put("trendGranularity", query.trendGranularity());
result.put("exclusions", query.exclusions()); result.put("exclusions", query.exclusions());
result.put("extraFilters", query.extraFilters()); result.put("extraFilters", query.extraFilters());
if(query.extraFilters().containsKey(cn.yunsheng.quality.filters.SalesOrderTypes.KEY)||query.extraFilters().containsKey(cn.yunsheng.quality.filters.SalesOrderTypes.EXCLUDE_KEY))result.put("salesOrderTypeVersionId",cn.yunsheng.quality.filters.SalesOrderTypes.version(jdbc));
if(query.stage()!=null)result.put("stage",query.stage()); if(query.stage()!=null)result.put("stage",query.stage());
if(!query.columnFilters().isEmpty())result.put("columnFilters",query.columnFilters()); if(!query.columnFilters().isEmpty())result.put("columnFilters",query.columnFilters());
return result; return result;
@@ -0,0 +1,18 @@
package cn.yunsheng.quality.metrics;
import cn.yunsheng.quality.filters.SalesOrderTypes;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Transactional;
@Component
public class SalesOrderMetricBackfill implements ApplicationRunner {
private final JdbcTemplate jdbc;
public SalesOrderMetricBackfill(JdbcTemplate jdbc) { this.jdbc=jdbc; }
@Override @Transactional
public void run(ApplicationArguments args) {
new SalesOrderMetricRollups(jdbc).prepareCurrentRelease(SalesOrderTypes.version(jdbc));
}
}
@@ -0,0 +1,78 @@
package cn.yunsheng.quality.metrics;
import cn.yunsheng.quality.filters.QueryFilterFields;
import cn.yunsheng.quality.filters.SalesOrderTypes;
import java.util.*;
import org.springframework.jdbc.core.JdbcTemplate;
/** Immutable aggregates keyed by both mapping version and source build. Missing snapshots use facts. */
public final class SalesOrderMetricRollups {
private final JdbcTemplate jdbc;
public SalesOrderMetricRollups(JdbcTemplate jdbc) { this.jdbc = jdbc; }
public void prepareCurrentRelease(String version) {
if (version == null || version.isBlank()) return;
for (String build : jdbc.queryForList("""
SELECT DISTINCT rm.transform_build_id FROM current_release cr
JOIN release_member rm ON rm.release_id=cr.release_id
WHERE cr.dataset_id='quality'
""", String.class)) prepareBuild(version, build);
}
public void refreshCurrentBuild(String build) {
jdbc.update("DELETE FROM sales_order_metric_rollup_build WHERE build_id=?", build);
jdbc.update("DELETE FROM sales_order_metric_rollup WHERE build_id=?", build);
prepareBuild(SalesOrderTypes.version(jdbc), build);
}
public void prepareBuild(String version, String build) {
if (version == null || version.isBlank()) return;
// The version row serializes preparers, including startup backfill and imports.
jdbc.queryForList("SELECT id FROM sales_order_type_version WHERE id=? FOR UPDATE", String.class, version);
if (Boolean.TRUE.equals(jdbc.queryForObject("SELECT COUNT(*)>0 FROM sales_order_metric_rollup_build WHERE version_id=? AND build_id=?", Boolean.class, version, build))) return;
jdbc.update("DELETE FROM sales_order_metric_rollup WHERE version_id=? AND build_id=?", version, build);
for (String stage : List.of("black", "bright")) {
String table = "fact_" + stage;
String sql = MetricsRollupService.rollupInsertSql(table)
.replace("INSERT INTO metric_rollup (", "INSERT INTO sales_order_metric_rollup (version_id, order_type,")
.replace("SELECT f.build_id, ?", "SELECT ?, COALESCE(sot.order_type,'__EMPTY__'), f.build_id, ?")
.replace("FROM " + table + " f", "FROM " + table + " f LEFT JOIN sales_order_type_entry sot ON sot.version_id=? AND sot.order_no=f.sales_order_no")
.replace("GROUP BY f.build_id,", "GROUP BY sot.order_type, f.build_id,");
jdbc.update(sql, version, stage, version, build);
}
jdbc.update("""
INSERT INTO sales_order_metric_rollup_build(version_id,build_id,source_row_count)
SELECT ?,?,COALESCE(SUM(source_row_count),0) FROM sales_order_metric_rollup WHERE version_id=? AND build_id=?
""", version, build, version, build);
}
public boolean ready(String release, Map<String,List<String>> filters, boolean hasColumnFilters, Set<String> allowed) {
if (hasColumnFilters || filters.isEmpty() || !Set.of(SalesOrderTypes.KEY,SalesOrderTypes.EXCLUDE_KEY).containsAll(filters.keySet())) return false;
if (!SalesOrderTypes.allowed(allowed)) throw QueryFilterFields.bad("没有查看销售订单类型的权限");
validate(filters);
String version = SalesOrderTypes.version(jdbc);
if (version.isBlank()) return false;
Integer missing = jdbc.queryForObject("""
SELECT COUNT(*) FROM release_member rm LEFT JOIN sales_order_metric_rollup_build rb
ON rb.build_id=rm.transform_build_id AND rb.version_id=?
WHERE rm.release_id=? AND rb.build_id IS NULL
""", Integer.class, version, release);
return missing != null && missing == 0;
}
static void validate(Map<String,List<String>> filters) {
for (var values : filters.values()) for (String value : values)
if (!SalesOrderTypes.TYPES.contains(value) && !QueryFilterFields.EMPTY.equals(value)) throw QueryFilterFields.bad("销售订单类型无效");
}
public void append(StringBuilder where, List<Object> args, Map<String,List<String>> filters) {
validate(filters);
where.append(" AND f.version_id=?"); args.add(SalesOrderTypes.version(jdbc));
for (var entry : filters.entrySet()) {
if (entry.getValue().isEmpty()) continue;
where.append(" AND f.order_type ").append(SalesOrderTypes.EXCLUDE_KEY.equals(entry.getKey()) ? "NOT IN (" : "IN (")
.append(String.join(",",Collections.nCopies(entry.getValue().size(),"?"))).append(')');
args.addAll(entry.getValue());
}
}
}
@@ -33,7 +33,7 @@ public class OptionsController {
/** Reference names support maintenance without granting aggregate-query or source-detail access. */ /** Reference names support maintenance without granting aggregate-query or source-detail access. */
@GetMapping("/reference-names") @GetMapping("/reference-names")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','RULE_MAINTAINER','QUALITY_VIEWER','BUSINESS_ANALYST','AUDIT_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> referenceNames(@AuthenticationPrincipal AppUserPrincipal actor) { public Map<String, Object> referenceNames(@AuthenticationPrincipal AppUserPrincipal actor) {
Map<String, Object> available = options(null, actor); Map<String, Object> available = options(null, actor);
Map<String, Object> result = new LinkedHashMap<>(); Map<String, Object> result = new LinkedHashMap<>();
@@ -43,7 +43,7 @@ public class OptionsController {
} }
@GetMapping @GetMapping
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> options(@RequestParam(required = false) String releaseId, public Map<String, Object> options(@RequestParam(required = false) String releaseId,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
String selected = releases.resolvePublishedRelease(releaseId); String selected = releases.resolvePublishedRelease(releaseId);
@@ -127,6 +127,11 @@ public class OptionsController {
return response; return response;
} }
/** Rechecks that the release behind a saved result is still published. */
public void requirePublishedRelease(String releaseId) {
releases.resolvePublishedRelease(releaseId);
}
@SuppressWarnings("unchecked") @SuppressWarnings("unchecked")
private Map<String, Object> mappingDefinition(String releaseId) { private Map<String, Object> mappingDefinition(String releaseId) {
String raw = jdbc.queryForObject("SELECT CAST(m.definition AS CHAR) FROM data_release r JOIN mapping_version m ON m.id=r.mapping_version WHERE r.id=?", String.class, releaseId); String raw = jdbc.queryForObject("SELECT CAST(m.definition AS CHAR) FROM data_release r JOIN mapping_version m ON m.id=r.mapping_version WHERE r.id=?", String.class, releaseId);
@@ -3,6 +3,7 @@ package cn.yunsheng.quality.records;
import cn.yunsheng.quality.common.DataScope; import cn.yunsheng.quality.common.DataScope;
import cn.yunsheng.quality.common.ImmutableQueryCache; import cn.yunsheng.quality.common.ImmutableQueryCache;
import cn.yunsheng.quality.filters.QueryFilterFields; import cn.yunsheng.quality.filters.QueryFilterFields;
import cn.yunsheng.quality.filters.SalesOrderTypes;
import cn.yunsheng.quality.common.DepartmentLabels; import cn.yunsheng.quality.common.DepartmentLabels;
import cn.yunsheng.quality.common.SelectionValues; import cn.yunsheng.quality.common.SelectionValues;
import cn.yunsheng.quality.identity.AppUserPrincipal; import cn.yunsheng.quality.identity.AppUserPrincipal;
@@ -52,6 +53,10 @@ public class RecordController {
private final JsonMapper mapper; private final JsonMapper mapper;
private final GovernanceAccessService governance; private final GovernanceAccessService governance;
private final RecordsWorkbookExportService workbookExport; private final RecordsWorkbookExportService workbookExport;
private cn.yunsheng.quality.filters.SourceColumnQueries sourceColumns;
@org.springframework.beans.factory.annotation.Autowired
public void setSourceColumns(cn.yunsheng.quality.filters.SourceColumnQueries columns) { this.sourceColumns = columns; }
public RecordController(JdbcTemplate jdbc, ReleaseService releases, JsonMapper mapper, public RecordController(JdbcTemplate jdbc, ReleaseService releases, JsonMapper mapper,
GovernanceAccessService governance, RecordsWorkbookExportService workbookExport) { GovernanceAccessService governance, RecordsWorkbookExportService workbookExport) {
@@ -60,6 +65,7 @@ public class RecordController {
this.mapper = mapper; this.mapper = mapper;
this.governance = governance; this.governance = governance;
this.workbookExport = workbookExport; this.workbookExport = workbookExport;
this.sourceColumns = new cn.yunsheng.quality.filters.SourceColumnQueries(jdbc,governance,mapper);
} }
@@ -91,9 +97,12 @@ public class RecordController {
public Map<String,Object> records(String stage,Integer year,Integer month,Integer quarter,String department,String group,String product,String productSearch,String projectGroup,boolean includeUnassigned,String releaseId,int page,int pageSize,String field,String search,String sort,String direction,List<String> columns,String extraFilters,AppUserPrincipal actor) { public Map<String,Object> records(String stage,Integer year,Integer month,Integer quarter,String department,String group,String product,String productSearch,String projectGroup,boolean includeUnassigned,String releaseId,int page,int pageSize,String field,String search,String sort,String direction,List<String> columns,String extraFilters,AppUserPrincipal actor) {
return recordsFiltered(stage,year,month,quarter,department,group,product,productSearch,projectGroup,includeUnassigned,releaseId,page,pageSize,field,search,sort,direction,columns,extraFilters,null,actor); return recordsFiltered(stage,year,month,quarter,department,group,product,productSearch,projectGroup,includeUnassigned,releaseId,page,pageSize,field,search,sort,direction,columns,extraFilters,null,actor);
} }
public Map<String,Object> recordsFiltered(String stage,Integer year,Integer month,Integer quarter,String department,String group,String product,String productSearch,String projectGroup,boolean includeUnassigned,String releaseId,int page,int pageSize,String field,String search,String sort,String direction,List<String> columns,String extraFilters,String columnFilters,AppUserPrincipal actor) {
return recordsRangeFiltered(stage,year,month,quarter,department,group,product,productSearch,projectGroup,includeUnassigned,releaseId,page,pageSize,field,search,sort,direction,columns,extraFilters,columnFilters,null,null,actor);
}
@GetMapping @GetMapping
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> recordsFiltered(@RequestParam String stage, public Map<String, Object> recordsRangeFiltered(@RequestParam String stage,
@RequestParam(required = false) Integer year, @RequestParam(required = false) Integer year,
@RequestParam(required = false) Integer month, @RequestParam(required = false) Integer month,
@RequestParam(required = false) Integer quarter, @RequestParam(required = false) Integer quarter,
@@ -113,11 +122,13 @@ public class RecordController {
@RequestParam(required = false) List<String> columns, @RequestParam(required = false) List<String> columns,
@RequestParam(required=false) String extraFilters, @RequestParam(required=false) String extraFilters,
@RequestParam(required=false) String columnFilters, @RequestParam(required=false) String columnFilters,
@RequestParam(required=false) Integer monthStart,
@RequestParam(required=false) Integer monthEnd,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
governance.requireDetail(actor); governance.requireDetail(actor);
governance.requireBoundAction("QUERY", "query.detail"); governance.requireBoundAction("QUERY", "query.detail");
if (!List.of("black", "bright").contains(stage)) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "stage 必须是 black 或 bright"); if (!List.of("black", "bright").contains(stage)) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "stage 必须是 black 或 bright");
validatePeriodFilters(year, month, quarter); validatePeriodFilters(year, month, quarter, monthStart, monthEnd);
if (page < 1 || pageSize < 1 || pageSize > 200) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "分页参数无效"); if (page < 1 || pageSize < 1 || pageSize > 200) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "分页参数无效");
if (productSearch != null && productSearch.length() > SelectionValues.MAX_VALUE_LENGTH) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "产品搜索文本过长"); if (productSearch != null && productSearch.length() > SelectionValues.MAX_VALUE_LENGTH) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "产品搜索文本过长");
List<String> selectedColumns = validateColumns(columns, actor); List<String> selectedColumns = validateColumns(columns, actor);
@@ -134,7 +145,7 @@ public class RecordController {
StringBuilder predicate = new StringBuilder(" WHERE rm.release_id = ?"); StringBuilder predicate = new StringBuilder(" WHERE rm.release_id = ?");
List<Object> args = new ArrayList<>(); List<Object> args = new ArrayList<>();
args.add(selectedRelease); args.add(selectedRelease);
appendPeriodFilters(predicate, args, year, month, quarter); appendPeriodFilters(predicate, args, year, month, quarter, monthStart, monthEnd);
SelectionValues.appendPredicate(predicate, args, "f.department", department, DepartmentLabels::queryValue); SelectionValues.appendPredicate(predicate, args, "f.department", department, DepartmentLabels::queryValue);
SelectionValues.appendPredicate(predicate, args, "f.group_name", group, value -> value); SelectionValues.appendPredicate(predicate, args, "f.group_name", group, value -> value);
SelectionValues.appendPredicate(predicate, args, "f.product", product, value -> value); SelectionValues.appendPredicate(predicate, args, "f.product", product, value -> value);
@@ -150,10 +161,10 @@ public class RecordController {
args.add(search); args.add(search);
args.add(jsonPath(selectedField)); args.add(jsonPath(selectedField));
} }
new QueryFilterFields(jdbc,governance).append(predicate,args,stage,QueryFilterFields.parse(extraFilters),actor); new QueryFilterFields(jdbc,governance).append(predicate,args,stage,QueryFilterFields.parse(extraFilters),actor,selectedRelease);
var sourceFilters=cn.yunsheng.quality.filters.SourceColumnQueries.parse(columnFilters); var sourceFilters=cn.yunsheng.quality.filters.SourceColumnQueries.parse(columnFilters);
for(var rule:sourceFilters)if(!List.of(stage).containsAll((List<?>)rule.get("stages")))throw new ResponseStatusException(HttpStatus.BAD_REQUEST,"来源字段筛选必须属于当前阶段"); for(var rule:sourceFilters)if(!List.of(stage).containsAll((List<?>)rule.get("stages")))throw new ResponseStatusException(HttpStatus.BAD_REQUEST,"来源字段筛选必须属于当前阶段");
if(!sourceFilters.isEmpty())new cn.yunsheng.quality.filters.SourceColumnQueries(jdbc,governance,mapper).append(predicate,args,stage,sourceFilters,selectedRelease,actor); if(!sourceFilters.isEmpty())sourceColumns.append(predicate,args,stage,sourceFilters,selectedRelease,actor);
DataScope.append(predicate, args, actor, "f.department COLLATE utf8mb4_0900_as_cs", "f.group_name COLLATE utf8mb4_0900_as_cs"); DataScope.append(predicate, args, actor, "f.department COLLATE utf8mb4_0900_as_cs", "f.group_name COLLATE utf8mb4_0900_as_cs");
String countSql = """ String countSql = """
@@ -187,17 +198,20 @@ public class RecordController {
} }
// Field permissions remain current on every request; no authorized row JSON is cached. // Field permissions remain current on every request; no authorized row JSON is cached.
Set<String> allowedFields = governance.allowedFields(actor); Set<String> allowedFields = governance.allowedFields(actor);
List<Object> rowArgs = new ArrayList<>(args); rowArgs.addAll(ids); List<Object> rowArgs = new ArrayList<>();
String orderTypeProjection=SalesOrderTypes.allowed(allowedFields)
? SalesOrderTypes.expression(stage,SalesOrderTypes.version(jdbc),rowArgs) : "NULL";
rowArgs.addAll(args); rowArgs.addAll(ids);
List<Map<String, Object>> fetched = ids.isEmpty() ? List.of() : jdbc.query(""" List<Map<String, Object>> fetched = ids.isEmpty() ? List.of() : jdbc.query("""
SELECT f.id, sr.id AS source_revision_id, sr.source_file, f.sheet_name, f.physical_row_no, f.month_key, SELECT f.id, sr.id AS source_revision_id, sr.source_file, f.sheet_name, f.physical_row_no, f.month_key,
f.department, f.group_name, f.project_group, f.product, f.qualified_weight, f.scrap_weight, f.department, f.group_name, f.project_group, f.product, f.qualified_weight, f.scrap_weight,
f.total_weight, f.raw_values, f.components f.total_weight, f.raw_values, f.components, %s AS sales_order_type
FROM release_member rm FROM release_member rm
JOIN source_revision sr ON sr.id = rm.source_revision_id JOIN source_revision sr ON sr.id = rm.source_revision_id
JOIN %s f ON f.build_id = rm.transform_build_id JOIN %s f ON f.build_id = rm.transform_build_id
%s %s
AND f.id IN (%s) AND f.id IN (%s)
""".formatted(fact, predicate, String.join(",", java.util.Collections.nCopies(ids.size(), "?"))), """.formatted(orderTypeProjection, fact, predicate, String.join(",", java.util.Collections.nCopies(ids.size(), "?"))),
(rs, row) -> mapRow(rs, stage, allowedFields, selectedColumns), rowArgs.toArray()); (rs, row) -> mapRow(rs, stage, allowedFields, selectedColumns), rowArgs.toArray());
Map<String, Map<String, Object>> byId = new LinkedHashMap<>(); Map<String, Map<String, Object>> byId = new LinkedHashMap<>();
fetched.forEach(row -> byId.put(String.valueOf(row.get("id")), row)); fetched.forEach(row -> byId.put(String.valueOf(row.get("id")), row));
@@ -209,9 +223,12 @@ public class RecordController {
public ResponseEntity<StreamingResponseBody> export(Integer year,Integer month,Integer quarter,String department,String group,String product,String productSearch,String projectGroup,boolean includeUnassigned,String releaseId,String extraFilters,AppUserPrincipal actor) { public ResponseEntity<StreamingResponseBody> export(Integer year,Integer month,Integer quarter,String department,String group,String product,String productSearch,String projectGroup,boolean includeUnassigned,String releaseId,String extraFilters,AppUserPrincipal actor) {
return exportFiltered(year,month,quarter,department,group,product,productSearch,projectGroup,includeUnassigned,releaseId,extraFilters,null,null,actor); return exportFiltered(year,month,quarter,department,group,product,productSearch,projectGroup,includeUnassigned,releaseId,extraFilters,null,null,actor);
} }
public ResponseEntity<StreamingResponseBody> exportFiltered(Integer year,Integer month,Integer quarter,String department,String group,String product,String productSearch,String projectGroup,boolean includeUnassigned,String releaseId,String extraFilters,String stage,String columnFilters,AppUserPrincipal actor) {
return exportRangeFiltered(year,month,quarter,department,group,product,productSearch,projectGroup,includeUnassigned,releaseId,extraFilters,stage,columnFilters,null,null,actor);
}
@GetMapping("/export") @GetMapping("/export")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public ResponseEntity<StreamingResponseBody> exportFiltered(@RequestParam(required = false) Integer year, public ResponseEntity<StreamingResponseBody> exportRangeFiltered(@RequestParam(required = false) Integer year,
@RequestParam(required = false) Integer month, @RequestParam(required = false) Integer month,
@RequestParam(required = false) Integer quarter, @RequestParam(required = false) Integer quarter,
@RequestParam(required = false) String department, @RequestParam(required = false) String department,
@@ -224,12 +241,14 @@ public class RecordController {
@RequestParam(required=false) String extraFilters, @RequestParam(required=false) String extraFilters,
@RequestParam(required=false) String stage, @RequestParam(required=false) String stage,
@RequestParam(required=false) String columnFilters, @RequestParam(required=false) String columnFilters,
@RequestParam(required=false) Integer monthStart,
@RequestParam(required=false) Integer monthEnd,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
governance.requireDetail(actor); governance.requireDetail(actor);
governance.requireBoundAction("QUERY", "query.detail"); governance.requireBoundAction("QUERY", "query.detail");
if (productSearch != null && productSearch.length() > SelectionValues.MAX_VALUE_LENGTH) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "产品搜索文本过长"); if (productSearch != null && productSearch.length() > SelectionValues.MAX_VALUE_LENGTH) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "产品搜索文本过长");
Path file = stage==null&&columnFilters==null ? workbookExport.export(year,month,quarter,department,group,product,projectGroup,includeUnassigned,releaseId,actor,QueryFilterFields.parse(extraFilters),productSearch) : workbookExport.exportFiltered(year, month, quarter, department, group, product, projectGroup, Path file = workbookExport.exportRangeFiltered(year, month, quarter, department, group, product, projectGroup,
includeUnassigned, releaseId, actor, QueryFilterFields.parse(extraFilters), productSearch,stage,cn.yunsheng.quality.filters.SourceColumnQueries.parse(columnFilters)); includeUnassigned, releaseId, actor, QueryFilterFields.parse(extraFilters), productSearch,stage,cn.yunsheng.quality.filters.SourceColumnQueries.parse(columnFilters),monthStart,monthEnd);
StreamingResponseBody body = output -> { StreamingResponseBody body = output -> {
try (InputStream input = Files.newInputStream(file)) { try (InputStream input = Files.newInputStream(file)) {
input.transferTo(output); input.transferTo(output);
@@ -246,7 +265,11 @@ public class RecordController {
} }
static void appendPeriodFilters(StringBuilder predicate, List<Object> args, Integer year, Integer month, Integer quarter) { static void appendPeriodFilters(StringBuilder predicate, List<Object> args, Integer year, Integer month, Integer quarter) {
validatePeriodFilters(year, month, quarter); appendPeriodFilters(predicate,args,year,month,quarter,null,null);
}
static void appendPeriodFilters(StringBuilder predicate, List<Object> args, Integer year, Integer month, Integer quarter, Integer monthStart, Integer monthEnd) {
validatePeriodFilters(year, month, quarter, monthStart, monthEnd);
if (year != null) { predicate.append(" AND f.month_key LIKE ?"); args.add(String.format("%04d-%%", year)); } if (year != null) { predicate.append(" AND f.month_key LIKE ?"); args.add(String.format("%04d-%%", year)); }
if (month != null) { predicate.append(" AND RIGHT(f.month_key, 2) = ?"); args.add(String.format("%02d", month)); } if (month != null) { predicate.append(" AND RIGHT(f.month_key, 2) = ?"); args.add(String.format("%02d", month)); }
if (quarter != null) { if (quarter != null) {
@@ -255,13 +278,25 @@ public class RecordController {
args.add(firstMonth); args.add(firstMonth);
args.add(firstMonth + 2); args.add(firstMonth + 2);
} }
if (monthStart != null) {
predicate.append(" AND CAST(RIGHT(f.month_key, 2) AS UNSIGNED) BETWEEN ? AND ?");
args.add(monthStart); args.add(monthEnd);
}
} }
static void validatePeriodFilters(Integer year, Integer month, Integer quarter) { static void validatePeriodFilters(Integer year, Integer month, Integer quarter) {
validatePeriodFilters(year,month,quarter,null,null);
}
static void validatePeriodFilters(Integer year, Integer month, Integer quarter, Integer monthStart, Integer monthEnd) {
if (year != null && (year < 2000 || year > 2100)) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "年份无效"); if (year != null && (year < 2000 || year > 2100)) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "年份无效");
if (month != null && (month < 1 || month > 12)) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "月份无效"); if (month != null && (month < 1 || month > 12)) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "月份无效");
if (quarter != null && (quarter < 1 || quarter > 4)) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "季度无效"); if (quarter != null && (quarter < 1 || quarter > 4)) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "季度无效");
if (month != null && quarter != null) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "月份与季度筛选不可同时指定"); if (month != null && quarter != null) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "月份与季度筛选不可同时指定");
if ((monthStart == null) != (monthEnd == null)) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "monthStart 和 monthEnd 必须成对提供");
if (monthStart != null && (year == null || month != null || quarter != null
|| monthStart < 1 || monthStart > 12 || monthEnd < 1 || monthEnd > 12 || monthStart > monthEnd))
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "月份范围无效,且不能与 month 或 quarter 并用");
} }
private Map<String, Object> mapRow(ResultSet rs, String stage, Set<String> allowedFields, List<String> selectedColumns) throws SQLException { private Map<String, Object> mapRow(ResultSet rs, String stage, Set<String> allowedFields, List<String> selectedColumns) throws SQLException {
@@ -280,6 +315,7 @@ public class RecordController {
row.put("projectGroup", rs.getString("project_group")); row.put("projectGroup", rs.getString("project_group"));
row.put("group", rs.getString("group_name")); row.put("group", rs.getString("group_name"));
row.put("product", rs.getString("product")); row.put("product", rs.getString("product"));
if(SalesOrderTypes.allowed(allowedFields))row.put("salesOrderType",rs.getString("sales_order_type"));
Map<String, Object> visibleRaw = filterFields(readJson(rs.getString("raw_values")), allowedFields); Map<String, Object> visibleRaw = filterFields(readJson(rs.getString("raw_values")), allowedFields);
if (selectedColumns != null) { if (selectedColumns != null) {
Map<String, Object> selectedRaw = new LinkedHashMap<>(); Map<String, Object> selectedRaw = new LinkedHashMap<>();
@@ -2,6 +2,7 @@ package cn.yunsheng.quality.records;
import cn.yunsheng.quality.common.DataScope; import cn.yunsheng.quality.common.DataScope;
import cn.yunsheng.quality.filters.QueryFilterFields; import cn.yunsheng.quality.filters.QueryFilterFields;
import cn.yunsheng.quality.filters.SalesOrderTypes;
import cn.yunsheng.quality.common.DepartmentLabels; import cn.yunsheng.quality.common.DepartmentLabels;
import cn.yunsheng.quality.common.SelectionValues; import cn.yunsheng.quality.common.SelectionValues;
import cn.yunsheng.quality.governance.GovernanceAccessService; import cn.yunsheng.quality.governance.GovernanceAccessService;
@@ -51,6 +52,10 @@ public class RecordsWorkbookExportService {
private final ReleaseService releases; private final ReleaseService releases;
private final GovernanceAccessService governance; private final GovernanceAccessService governance;
private final JsonMapper mapper; private final JsonMapper mapper;
private cn.yunsheng.quality.filters.SourceColumnQueries sourceColumns;
@org.springframework.beans.factory.annotation.Autowired
public void setSourceColumns(cn.yunsheng.quality.filters.SourceColumnQueries columns) { this.sourceColumns = columns; }
public RecordsWorkbookExportService(JdbcTemplate jdbc, ReleaseService releases, public RecordsWorkbookExportService(JdbcTemplate jdbc, ReleaseService releases,
GovernanceAccessService governance, JsonMapper mapper) { GovernanceAccessService governance, JsonMapper mapper) {
@@ -58,6 +63,7 @@ public class RecordsWorkbookExportService {
this.releases = releases; this.releases = releases;
this.governance = governance; this.governance = governance;
this.mapper = mapper; this.mapper = mapper;
this.sourceColumns = new cn.yunsheng.quality.filters.SourceColumnQueries(jdbc,governance,mapper);
} }
public Path export(Integer year, Integer month, Integer quarter, String department, String group, public Path export(Integer year, Integer month, Integer quarter, String department, String group,
@@ -72,10 +78,13 @@ public class RecordsWorkbookExportService {
return exportFiltered(year,month,quarter,department,group,product,projectGroup,includeUnassigned,releaseId,actor,extraFilters,productSearch,null,List.of()); return exportFiltered(year,month,quarter,department,group,product,projectGroup,includeUnassigned,releaseId,actor,extraFilters,productSearch,null,List.of());
} }
public Path exportFiltered(Integer year,Integer month,Integer quarter,String department,String group,String product,String projectGroup,boolean includeUnassigned,String releaseId,AppUserPrincipal actor,Map<String,List<String>> extraFilters,String productSearch,String selectedStage,List<Map<String,Object>> columnFilters) { public Path exportFiltered(Integer year,Integer month,Integer quarter,String department,String group,String product,String projectGroup,boolean includeUnassigned,String releaseId,AppUserPrincipal actor,Map<String,List<String>> extraFilters,String productSearch,String selectedStage,List<Map<String,Object>> columnFilters) {
return exportRangeFiltered(year,month,quarter,department,group,product,projectGroup,includeUnassigned,releaseId,actor,extraFilters,productSearch,selectedStage,columnFilters,null,null);
}
public Path exportRangeFiltered(Integer year,Integer month,Integer quarter,String department,String group,String product,String projectGroup,boolean includeUnassigned,String releaseId,AppUserPrincipal actor,Map<String,List<String>> extraFilters,String productSearch,String selectedStage,List<Map<String,Object>> columnFilters,Integer monthStart,Integer monthEnd) {
if(selectedStage!=null&&!List.of("black","bright").contains(selectedStage))throw new ResponseStatusException(HttpStatus.BAD_REQUEST,"stage 必须是 black 或 bright"); if(selectedStage!=null&&!List.of("black","bright").contains(selectedStage))throw new ResponseStatusException(HttpStatus.BAD_REQUEST,"stage 必须是 black 或 bright");
columnFilters=cn.yunsheng.quality.filters.SourceColumnQueries.parse(columnFilters); columnFilters=cn.yunsheng.quality.filters.SourceColumnQueries.parse(columnFilters);
if(selectedStage!=null)for(var rule:columnFilters)if(!List.of(selectedStage).containsAll((List<?>)rule.get("stages")))throw new ResponseStatusException(HttpStatus.BAD_REQUEST,"来源字段筛选必须属于当前阶段"); if(selectedStage!=null)for(var rule:columnFilters)if(!List.of(selectedStage).containsAll((List<?>)rule.get("stages")))throw new ResponseStatusException(HttpStatus.BAD_REQUEST,"来源字段筛选必须属于当前阶段");
RecordController.validatePeriodFilters(year, month, quarter); RecordController.validatePeriodFilters(year, month, quarter, monthStart, monthEnd);
if (productSearch != null && productSearch.length() > SelectionValues.MAX_VALUE_LENGTH) if (productSearch != null && productSearch.length() > SelectionValues.MAX_VALUE_LENGTH)
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "产品搜索文本过长"); throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "产品搜索文本过长");
String selectedRelease = releases.resolvePublishedRelease(releaseId); String selectedRelease = releases.resolvePublishedRelease(releaseId);
@@ -88,9 +97,9 @@ public class RecordsWorkbookExportService {
workbook.setCompressTempFiles(true); workbook.setCompressTempFiles(true);
CellStyle headerStyle = createHeaderStyle(workbook); CellStyle headerStyle = createHeaderStyle(workbook);
if(selectedStage==null||selectedStage.equals("black"))createStageSheet(workbook, headerStyle, "黑片", "fact_black", selectedRelease, year, month, quarter, if(selectedStage==null||selectedStage.equals("black"))createStageSheet(workbook, headerStyle, "黑片", "fact_black", selectedRelease, year, month, quarter,
department, group, product, projectGroup, includeUnassigned, allowedFields, actor, extraFilters, productSearch,columnFilters); department, group, product, projectGroup, includeUnassigned, allowedFields, actor, extraFilters, productSearch,columnFilters,monthStart,monthEnd);
if(selectedStage==null||selectedStage.equals("bright"))createStageSheet(workbook, headerStyle, "亮片", "fact_bright", selectedRelease, year, month, quarter, if(selectedStage==null||selectedStage.equals("bright"))createStageSheet(workbook, headerStyle, "亮片", "fact_bright", selectedRelease, year, month, quarter,
department, group, product, projectGroup, includeUnassigned, allowedFields, actor, extraFilters, productSearch,columnFilters); department, group, product, projectGroup, includeUnassigned, allowedFields, actor, extraFilters, productSearch,columnFilters,monthStart,monthEnd);
try (var stream = Files.newOutputStream(output)) { try (var stream = Files.newOutputStream(output)) {
workbook.write(stream); workbook.write(stream);
} }
@@ -111,18 +120,23 @@ public class RecordsWorkbookExportService {
String releaseId, Integer year, Integer month, Integer quarter, String department, String releaseId, Integer year, Integer month, Integer quarter, String department,
String group, String product, String projectGroup, boolean includeUnassigned, String group, String product, String projectGroup, boolean includeUnassigned,
Set<String> allowedFields, AppUserPrincipal actor, Map<String,List<String>> extraFilters, Set<String> allowedFields, AppUserPrincipal actor, Map<String,List<String>> extraFilters,
String productSearch,List<Map<String,Object>> columnFilters) throws IOException { String productSearch,List<Map<String,Object>> columnFilters,Integer monthStart,Integer monthEnd) throws IOException {
Sheet sheet = workbook.createSheet(sheetName); Sheet sheet = workbook.createSheet(sheetName);
LinkedHashSet<String> rawFields = new LinkedHashSet<>(); LinkedHashSet<String> rawFields = new LinkedHashSet<>();
if (releaseId != null) { if (releaseId != null) {
QueryScope rawScope = queryScope(fact, releaseId, year, month, quarter, department, group, product, QueryScope rawScope = queryScope(fact, releaseId, year, month, quarter, department, group, product,
projectGroup, includeUnassigned, "f.raw_values", actor, extraFilters, productSearch,columnFilters); projectGroup, includeUnassigned, "CAST(JSON_KEYS(f.raw_values) AS CHAR)", actor, extraFilters, productSearch,columnFilters,monthStart,monthEnd);
scan(rawScope, rs -> rawFields.addAll(visibleRaw(readJson(rs.getString(1)), allowedFields).keySet())); scan(rawScope, rs -> {
Object decoded=mapper.readValue(rs.getString(1),Object.class);
if(decoded instanceof List<?> keys)for(Object key:keys)if(key instanceof String name&&(allowedFields.contains("*")||allowedFields.contains(name)))rawFields.add(name);
});
} }
if (rawFields.size() + HEADER_COUNT > MAX_COLUMNS) boolean showOrderType=SalesOrderTypes.allowed(allowedFields);
if (rawFields.size() + HEADER_COUNT + (showOrderType?1:0) > MAX_COLUMNS)
throw new ResponseStatusException(HttpStatus.PAYLOAD_TOO_LARGE, "原始字段数量超过 Excel 单表列数上限"); throw new ResponseStatusException(HttpStatus.PAYLOAD_TOO_LARGE, "原始字段数量超过 Excel 单表列数上限");
List<String> headers = new ArrayList<>(FIXED_HEADERS); List<String> headers = new ArrayList<>(FIXED_HEADERS);
if(showOrderType)headers.add(SalesOrderTypes.LABEL);
headers.addAll(rawFields); headers.addAll(rawFields);
Row header = sheet.createRow(0); Row header = sheet.createRow(0);
for (int index = 0; index < headers.size(); index++) { for (int index = 0; index < headers.size(); index++) {
@@ -138,19 +152,19 @@ public class RecordsWorkbookExportService {
QueryScope rowScope = queryScope(fact, releaseId, year, month, quarter, department, group, product, QueryScope rowScope = queryScope(fact, releaseId, year, month, quarter, department, group, product,
projectGroup, includeUnassigned, projectGroup, includeUnassigned,
"sr.source_file, f.sheet_name, f.physical_row_no, sr.id AS source_revision_id, f.month_key, " + "sr.source_file, f.sheet_name, f.physical_row_no, sr.id AS source_revision_id, f.month_key, " +
"f.department, f.group_name, f.project_group, f.product, f.raw_values", actor, extraFilters, productSearch,columnFilters); "f.department, f.group_name, f.project_group, f.product, f.raw_values", actor, extraFilters, productSearch,columnFilters,monthStart,monthEnd);
int[] rowIndex = {1}; int[] rowIndex = {1};
scan(rowScope, rs -> writeRow(sheet, rowIndex, rs, rawFields, allowedFields)); scan(rowScope, rs -> writeRow(sheet, rowIndex, rs, rawFields, allowedFields,showOrderType));
} }
private QueryScope queryScope(String fact, String releaseId, Integer year, Integer month, Integer quarter, private QueryScope queryScope(String fact, String releaseId, Integer year, Integer month, Integer quarter,
String department, String group, String product, String projectGroup, String department, String group, String product, String projectGroup,
boolean includeUnassigned, String select, AppUserPrincipal actor, Map<String,List<String>> extraFilters, boolean includeUnassigned, String select, AppUserPrincipal actor, Map<String,List<String>> extraFilters,
String productSearch,List<Map<String,Object>> columnFilters) { String productSearch,List<Map<String,Object>> columnFilters,Integer monthStart,Integer monthEnd) {
StringBuilder predicate = new StringBuilder(" WHERE rm.release_id = ?"); StringBuilder predicate = new StringBuilder(" WHERE rm.release_id = ?");
List<Object> args = new ArrayList<>(); List<Object> args = new ArrayList<>();
args.add(releaseId); args.add(releaseId);
RecordController.appendPeriodFilters(predicate, args, year, month, quarter); RecordController.appendPeriodFilters(predicate, args, year, month, quarter, monthStart, monthEnd);
SelectionValues.appendPredicate(predicate, args, "f.department", department, DepartmentLabels::queryValue); SelectionValues.appendPredicate(predicate, args, "f.department", department, DepartmentLabels::queryValue);
SelectionValues.appendPredicate(predicate, args, "f.group_name", group, value -> value); SelectionValues.appendPredicate(predicate, args, "f.group_name", group, value -> value);
SelectionValues.appendPredicate(predicate, args, "f.product", product, value -> value); SelectionValues.appendPredicate(predicate, args, "f.product", product, value -> value);
@@ -161,11 +175,16 @@ public class RecordsWorkbookExportService {
SelectionValues.appendPredicate(predicate, args, "f.project_group", projectGroup, value -> value); SelectionValues.appendPredicate(predicate, args, "f.project_group", projectGroup, value -> value);
if (!includeUnassigned && !SelectionValues.parse(group).contains("__UNASSIGNED__")) if (!includeUnassigned && !SelectionValues.parse(group).contains("__UNASSIGNED__"))
predicate.append(" AND (f.group_name IS NOT NULL AND f.group_name <> '')"); predicate.append(" AND (f.group_name IS NOT NULL AND f.group_name <> '')");
new QueryFilterFields(jdbc,governance).append(predicate,args,fact.equals("fact_black")?"black":"bright",extraFilters,actor); new QueryFilterFields(jdbc,governance).append(predicate,args,fact.equals("fact_black")?"black":"bright",extraFilters,actor,releaseId);
if(!columnFilters.isEmpty())new cn.yunsheng.quality.filters.SourceColumnQueries(jdbc,governance,mapper).append(predicate,args,fact.equals("fact_black")?"black":"bright",columnFilters,releaseId,actor); if(!columnFilters.isEmpty())sourceColumns.append(predicate,args,fact.equals("fact_black")?"black":"bright",columnFilters,releaseId,actor);
DataScope.append(predicate, args, actor, "f.department COLLATE utf8mb4_0900_as_cs", DataScope.append(predicate, args, actor, "f.department COLLATE utf8mb4_0900_as_cs",
"f.group_name COLLATE utf8mb4_0900_as_cs"); "f.group_name COLLATE utf8mb4_0900_as_cs");
String columns = select == null ? "f.raw_values" : select; String columns = select == null ? "f.raw_values" : select;
if(columns.startsWith("sr.source_file")&&SalesOrderTypes.allowed(governance.allowedFields(actor))) {
List<Object> projectionArgs=new ArrayList<>();
columns+=", "+SalesOrderTypes.expression(fact.equals("fact_black")?"black":"bright",SalesOrderTypes.version(jdbc),projectionArgs)+" AS sales_order_type";
projectionArgs.addAll(args);args=projectionArgs;
}
String sql = "SELECT " + columns + " FROM release_member rm " + String sql = "SELECT " + columns + " FROM release_member rm " +
"JOIN source_revision sr ON sr.id = rm.source_revision_id " + "JOIN source_revision sr ON sr.id = rm.source_revision_id " +
"JOIN " + fact + " f ON f.build_id = rm.transform_build_id " + predicate + "JOIN " + fact + " f ON f.build_id = rm.transform_build_id " + predicate +
@@ -184,7 +203,7 @@ public class RecordsWorkbookExportService {
} }
private void writeRow(Sheet sheet, int[] rowIndex, ResultSet rs, Set<String> rawFields, private void writeRow(Sheet sheet, int[] rowIndex, ResultSet rs, Set<String> rawFields,
Set<String> allowedFields) throws SQLException { Set<String> allowedFields,boolean showOrderType) throws SQLException {
if (rowIndex[0] >= MAX_ROWS) if (rowIndex[0] >= MAX_ROWS)
throw new ResponseStatusException(HttpStatus.PAYLOAD_TOO_LARGE, "明细超过 Excel 单工作表行数上限"); throw new ResponseStatusException(HttpStatus.PAYLOAD_TOO_LARGE, "明细超过 Excel 单工作表行数上限");
Row row = sheet.createRow(rowIndex[0]++); Row row = sheet.createRow(rowIndex[0]++);
@@ -201,6 +220,7 @@ public class RecordsWorkbookExportService {
Map<String, Object> raw = visibleRaw(readJson(rs.getString("raw_values")), allowedFields); Map<String, Object> raw = visibleRaw(readJson(rs.getString("raw_values")), allowedFields);
int index = HEADER_COUNT; int index = HEADER_COUNT;
if(showOrderType)writeText(row,index++,rs.getString("sales_order_type"));
for (String field : rawFields) { for (String field : rawFields) {
if (raw.containsKey(field)) writeText(row, index, valueText(raw.get(field))); if (raw.containsKey(field)) writeText(row, index, valueText(raw.get(field)));
index++; index++;
@@ -19,11 +19,11 @@ public class ReleaseController {
public ReleaseController(ReleaseService service) { this.service = service; } public ReleaseController(ReleaseService service) { this.service = service; }
@GetMapping @GetMapping
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST','DATA_MAINTAINER','DATA_PUBLISHER','AUDIT_VIEWER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> list() { return service.list(); } public Map<String, Object> list() { return service.list(); }
@PostMapping("/{id}/publish") @PostMapping("/{id}/publish")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_PUBLISHER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> publish(@PathVariable String id, @RequestBody ReleaseService.ReleaseActionRequest request, public Map<String, Object> publish(@PathVariable String id, @RequestBody ReleaseService.ReleaseActionRequest request,
@AuthenticationPrincipal AppUserPrincipal actor) { @AuthenticationPrincipal AppUserPrincipal actor) {
return service.transition(id, "publish", request, actor); return service.transition(id, "publish", request, actor);
@@ -92,7 +92,7 @@ public class ReleaseService {
public Map<String, Object> transitionInTestWorkflow(String releaseId, String action, ReleaseActionRequest request, public Map<String, Object> transitionInTestWorkflow(String releaseId, String action, ReleaseActionRequest request,
AppUserPrincipal actor, boolean allowEmptyRelease) { AppUserPrincipal actor, boolean allowEmptyRelease) {
if (!testWorkflowMode) throw new ResponseStatusException(HttpStatus.NOT_FOUND, "测试模式便捷工作流未启用"); if (!testWorkflowMode) throw new ResponseStatusException(HttpStatus.NOT_FOUND, "测试模式便捷工作流未启用");
if (!actor.getRoleCodes().contains("SYSTEM_ADMIN") && !actor.getRoleCodes().contains("DATA_PUBLISHER")) if (!actor.getRoleCodes().contains("SYSTEM_ADMIN") && !actor.getRoleCodes().contains("DATA_ADMIN"))
throw new ResponseStatusException(HttpStatus.FORBIDDEN, "自动更新报表需要发布者或系统管理员权限"); throw new ResponseStatusException(HttpStatus.FORBIDDEN, "自动更新报表需要发布者或系统管理员权限");
return transitionInternal(releaseId, action, request, actor, true, allowEmptyRelease); return transitionInternal(releaseId, action, request, actor, true, allowEmptyRelease);
} }
@@ -30,13 +30,13 @@ public class SourceController {
} }
@GetMapping @GetMapping
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','QUALITY_VIEWER','BUSINESS_ANALYST','DATA_MAINTAINER','DATA_PUBLISHER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN', 'USER')")
public Map<String, Object> list() { public Map<String, Object> list() {
return Map.of("items", catalog.list()); return Map.of("items", catalog.list());
} }
@PostMapping("/scan") @PostMapping("/scan")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> scan() { public Map<String, Object> scan() {
access.requireBoundAction("RECEIVER","source.receive"); access.requireBoundAction("RECEIVER","source.receive");
List<SourceCatalogService.SourceView> items = catalog.scan(); List<SourceCatalogService.SourceView> items = catalog.scan();
@@ -44,7 +44,7 @@ public class SourceController {
} }
@PostMapping("/upload") @PostMapping("/upload")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String, Object> upload(@RequestPart("file") MultipartFile file) { public Map<String, Object> upload(@RequestPart("file") MultipartFile file) {
access.requireBoundAction("RECEIVER","source.receive"); access.requireBoundAction("RECEIVER","source.receive");
if (file.isEmpty()) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "文件为空"); if (file.isEmpty()) throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "文件为空");
@@ -14,13 +14,13 @@ import org.springframework.web.server.ResponseStatusException;
public class SourceDeletionController { public class SourceDeletionController {
private final SourceDeletionService deletion; private final SourceDeletionService deletion;
public SourceDeletionController(SourceDeletionService deletion){this.deletion=deletion;} public SourceDeletionController(SourceDeletionService deletion){this.deletion=deletion;}
@GetMapping("/{id}/deletion-preview") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','DATA_PUBLISHER')") @GetMapping("/{id}/deletion-preview") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> preview(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor){return deletion.preview(id,actor);} public Map<String,Object> preview(@PathVariable String id,@AuthenticationPrincipal AppUserPrincipal actor){return deletion.preview(id,actor);}
@GetMapping("/deleted") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','DATA_PUBLISHER')") @GetMapping("/deleted") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> deleted(@AuthenticationPrincipal AppUserPrincipal actor){return deletion.deleted(actor);} public Map<String,Object> deleted(@AuthenticationPrincipal AppUserPrincipal actor){return deletion.deleted(actor);}
@PostMapping("/{id}/delete") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER')") @PostMapping("/{id}/delete") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> delete(@PathVariable String id,@RequestBody SourceDeletionService.DeleteRequest request,@AuthenticationPrincipal AppUserPrincipal actor){return deletion.delete(id,request,actor);} public Map<String,Object> delete(@PathVariable String id,@RequestBody SourceDeletionService.DeleteRequest request,@AuthenticationPrincipal AppUserPrincipal actor){return deletion.delete(id,request,actor);}
@PostMapping("/{id}/restore") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER')") @PostMapping("/{id}/restore") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public Map<String,Object> restore(@PathVariable String id,@RequestBody SourceDeletionService.RestoreRequest request,@AuthenticationPrincipal AppUserPrincipal actor){return deletion.restore(id,request,actor);} public Map<String,Object> restore(@PathVariable String id,@RequestBody SourceDeletionService.RestoreRequest request,@AuthenticationPrincipal AppUserPrincipal actor){return deletion.restore(id,request,actor);}
@ExceptionHandler(ConcurrencyFailureException.class) public org.springframework.http.ProblemDetail concurrent(){return org.springframework.http.ProblemDetail.forStatusAndDetail(HttpStatus.CONFLICT,"来源正在发生并发操作,请刷新预览后重试");} @ExceptionHandler(ConcurrencyFailureException.class) public org.springframework.http.ProblemDetail concurrent(){return org.springframework.http.ProblemDetail.forStatusAndDetail(HttpStatus.CONFLICT,"来源正在发生并发操作,请刷新预览后重试");}
} }
@@ -60,7 +60,7 @@ public class SourceDeletionService {
g.audit(actor.getId().toString(),"SOURCE_VISIBILITY_RESTORED","source_candidate",id,Map.of("reason",reason,"onlyVisibility",true,"physicalFilesRetained",true));return view(source(id,false),current,actor); g.audit(actor.getId().toString(),"SOURCE_VISIBILITY_RESTORED","source_candidate",id,Map.of("reason",reason,"onlyVisibility",true,"physicalFilesRetained",true));return view(source(id,false),current,actor);
} }
private Map<String,Object> view(Map<String,Object> row,String current,AppUserPrincipal actor){ private Map<String,Object> view(Map<String,Object> row,String current,AppUserPrincipal actor){
Map<String,Object> usage=currentUsage(row,current),change=change(row);boolean hidden=row.get("deletedAt")!=null,busy=busy(row),manage=has(actor,"SYSTEM_ADMIN","DATA_MAINTAINER"),publisher=has(actor,"SYSTEM_ADMIN","DATA_PUBLISHER"),receiver=receiverAllowed(); Map<String,Object> usage=currentUsage(row,current),change=change(row);boolean hidden=row.get("deletedAt")!=null,busy=busy(row),manage=has(actor,"SYSTEM_ADMIN", "DATA_ADMIN"),publisher=has(actor,"SYSTEM_ADMIN", "DATA_ADMIN"),receiver=receiverAllowed();
boolean buildScope=true;if(publisher&&activeChange(change,usage,current)&&"APPROVED".equals(change.get("status")))try{sourceAccess.requireReleaseSources(current,actor);}catch(ResponseStatusException denied){if(denied.getStatusCode()!=HttpStatus.FORBIDDEN)throw denied;buildScope=false;} boolean buildScope=true;if(publisher&&activeChange(change,usage,current)&&"APPROVED".equals(change.get("status")))try{sourceAccess.requireReleaseSources(current,actor);}catch(ResponseStatusException denied){if(denied.getStatusCode()!=HttpStatus.FORBIDDEN)throw denied;buildScope=false;}
boolean active=activeChange(change,usage,current),review=active&&"REQUESTED".equals(change.get("status"))&&publisher&&!busy,build=active&&"APPROVED".equals(change.get("status"))&&publisher&&buildScope&&!busy; boolean active=activeChange(change,usage,current),review=active&&"REQUESTED".equals(change.get("status"))&&publisher&&!busy,build=active&&"APPROVED".equals(change.get("status"))&&publisher&&buildScope&&!busy;
boolean canDelete=manage&&receiver&&!hidden&&!busy&&!active,canRestore=manage&&receiver&&hidden&&!busy; boolean canDelete=manage&&receiver&&!hidden&&!busy&&!active,canRestore=manage&&receiver&&hidden&&!busy;
@@ -71,7 +71,7 @@ public class SourceDeletionService {
Map<String,Object> out=new LinkedHashMap<>();out.put("source",source);out.put("currentReleaseId",current);out.put("currentUsage",usage);out.put("currentUsed",usage!=null);out.put("documentId",usage==null?null:usage.get("documentId"));out.put("revisionId",usage==null?null:usage.get("revisionId"));out.put("mode",usage==null?"HIDE_ONLY":"WITHDRAW_CURRENT");out.put("history",history);out.put("deleteVersion",row.get("deleteVersion"));out.put("hidden",hidden);out.put("changeRequest",change);out.put("nextAction",next);out.put("canDelete",canDelete);out.put("canRestore",canRestore);out.put("canReview",review);out.put("canBuild",build);out.put("actions",Map.of("canRequest",canDelete,"canReview",review,"canBuild",build));out.put("blockedReason",blocked);out.put("physicalFilesRetained",true);out.put("currentReleaseUnchanged",true);return out; Map<String,Object> out=new LinkedHashMap<>();out.put("source",source);out.put("currentReleaseId",current);out.put("currentUsage",usage);out.put("currentUsed",usage!=null);out.put("documentId",usage==null?null:usage.get("documentId"));out.put("revisionId",usage==null?null:usage.get("revisionId"));out.put("mode",usage==null?"HIDE_ONLY":"WITHDRAW_CURRENT");out.put("history",history);out.put("deleteVersion",row.get("deleteVersion"));out.put("hidden",hidden);out.put("changeRequest",change);out.put("nextAction",next);out.put("canDelete",canDelete);out.put("canRestore",canRestore);out.put("canReview",review);out.put("canBuild",build);out.put("actions",Map.of("canRequest",canDelete,"canReview",review,"canBuild",build));out.put("blockedReason",blocked);out.put("physicalFilesRetained",true);out.put("currentReleaseUnchanged",true);return out;
} }
private AppUserPrincipal principal(AppUserPrincipal actor){access.requireUserActivePolicy(actor);return access.resolvePrincipal(actor);} private AppUserPrincipal principal(AppUserPrincipal actor){access.requireUserActivePolicy(actor);return access.resolvePrincipal(actor);}
private void requireManage(AppUserPrincipal actor){if(!has(actor,"SYSTEM_ADMIN","DATA_MAINTAINER"))throw error(HttpStatus.FORBIDDEN,"只有数据维护者可以申请删除或恢复来源");} private void requireManage(AppUserPrincipal actor){if(!has(actor,"SYSTEM_ADMIN", "DATA_ADMIN"))throw error(HttpStatus.FORBIDDEN,"只有数据维护者可以申请删除或恢复来源");}
private boolean has(AppUserPrincipal actor,String...roles){return actor.getRoleCodes().stream().anyMatch(Set.of(roles)::contains);} private boolean has(AppUserPrincipal actor,String...roles){return actor.getRoleCodes().stream().anyMatch(Set.of(roles)::contains);}
private void authorize(Map<String,Object> source,AppUserPrincipal actor){ private void authorize(Map<String,Object> source,AppUserPrincipal actor){
List<Map<String,Object>> revisions=revisions(source);for(var revision:revisions)sourceAccess.requireSource((String)revision.get("sourceDocumentId"),(String)revision.get("id"),actor); List<Map<String,Object>> revisions=revisions(source);for(var revision:revisions)sourceAccess.requireSource((String)revision.get("sourceDocumentId"),(String)revision.get("id"),actor);
@@ -12,11 +12,11 @@ import org.springframework.web.server.ResponseStatusException;
@RestController @RestController
@RequestMapping("/api/sources") @RequestMapping("/api/sources")
@PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER')") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')")
public class SourceUploadController { public class SourceUploadController {
private final SourceUploadService uploads; private final SourceUploadService uploads;
public SourceUploadController(SourceUploadService uploads){this.uploads=uploads;} public SourceUploadController(SourceUploadService uploads){this.uploads=uploads;}
@GetMapping("/upload-options") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN','DATA_MAINTAINER','DATA_PUBLISHER')") public Map<String,Object> options(@AuthenticationPrincipal AppUserPrincipal actor){return uploads.options(actor);} @GetMapping("/upload-options") @PreAuthorize("hasAnyRole('SYSTEM_ADMIN', 'DATA_ADMIN')") public Map<String,Object> options(@AuthenticationPrincipal AppUserPrincipal actor){return uploads.options(actor);}
@PostMapping("/workbook-preview") public Map<String,Object> previewUpload(@RequestPart("file") MultipartFile file,@AuthenticationPrincipal AppUserPrincipal actor){try{return uploads.previewUpload(file,actor);}catch(IOException|IllegalArgumentException e){throw new ResponseStatusException(HttpStatus.BAD_REQUEST,e.getMessage());}} @PostMapping("/workbook-preview") public Map<String,Object> previewUpload(@RequestPart("file") MultipartFile file,@AuthenticationPrincipal AppUserPrincipal actor){try{return uploads.previewUpload(file,actor);}catch(IOException|IllegalArgumentException e){throw new ResponseStatusException(HttpStatus.BAD_REQUEST,e.getMessage());}}
@GetMapping("/workbook-preview/{id}") public Map<String,Object> preview(@PathVariable String id,@RequestParam(required=false) String sheetName,@RequestParam(defaultValue="1") int headerRow,@AuthenticationPrincipal AppUserPrincipal actor){try{return uploads.preview(id,sheetName,headerRow,actor);}catch(IOException|IllegalArgumentException e){throw new ResponseStatusException(HttpStatus.BAD_REQUEST,e.getMessage());}} @GetMapping("/workbook-preview/{id}") public Map<String,Object> preview(@PathVariable String id,@RequestParam(required=false) String sheetName,@RequestParam(defaultValue="1") int headerRow,@AuthenticationPrincipal AppUserPrincipal actor){try{return uploads.preview(id,sheetName,headerRow,actor);}catch(IOException|IllegalArgumentException e){throw new ResponseStatusException(HttpStatus.BAD_REQUEST,e.getMessage());}}
@PostMapping("/configure") public Map<String,Object> configure(@RequestBody SourceUploadService.ConfigureRequest request,@AuthenticationPrincipal AppUserPrincipal actor){try{return uploads.configure(request,actor);}catch(IOException|IllegalArgumentException e){throw new ResponseStatusException(HttpStatus.BAD_REQUEST,e.getMessage());}} @PostMapping("/configure") public Map<String,Object> configure(@RequestBody SourceUploadService.ConfigureRequest request,@AuthenticationPrincipal AppUserPrincipal actor){try{return uploads.configure(request,actor);}catch(IOException|IllegalArgumentException e){throw new ResponseStatusException(HttpStatus.BAD_REQUEST,e.getMessage());}}
@@ -32,8 +32,8 @@ public class SourceUploadService {
} }
public Map<String,Object> options(AppUserPrincipal original){ public Map<String,Object> options(AppUserPrincipal original){
access.requireUserActivePolicy(original);AppUserPrincipal actor=access.resolvePrincipal(original);String current=current(false);List<Map<String,Object>> targets=new ArrayList<>(); access.requireUserActivePolicy(original);AppUserPrincipal actor=access.resolvePrincipal(original);String current=current(false);List<Map<String,Object>> targets=new ArrayList<>();
if(current!=null)for(Map<String,Object> row:g.rows("SELECT d.id AS document_id,r.id AS revision_id,d.source_slot,d.stage,d.source_year AS year,d.source_month AS month,d.scope_code,r.source_file,r.sha256,r.parser_config_hash FROM release_member m JOIN source_document d ON d.id=m.source_document_id JOIN source_revision r ON r.id=m.source_revision_id WHERE m.release_id=? ORDER BY d.source_year,d.source_month,d.stage",current)){ if(current!=null)for(Map<String,Object> row:g.rows("SELECT d.id AS document_id,r.id AS revision_id,d.source_slot,d.source_mode,d.stage,d.source_year AS year,d.source_month AS month,d.scope_code,r.source_file,r.sha256,r.parser_config_hash FROM release_member m JOIN source_document d ON d.id=m.source_document_id JOIN source_revision r ON r.id=m.source_revision_id WHERE m.release_id=? ORDER BY d.source_year,d.source_month,d.stage,d.source_slot",current)){
try{sourceAccess.requireSource((String)row.get("documentId"),(String)row.get("revisionId"),actor);targets.add(row);}catch(ResponseStatusException denied){if(denied.getStatusCode()!=HttpStatus.FORBIDDEN)throw denied;} try{sourceAccess.requireSource((String)row.get("documentId"),(String)row.get("revisionId"),actor);row.put("fileId",fileId(row.get("sourceSlot")));targets.add(row);}catch(ResponseStatusException denied){if(denied.getStatusCode()!=HttpStatus.FORBIDDEN)throw denied;}
} }
List<String> currentSourceIds=new ArrayList<>(); List<String> currentSourceIds=new ArrayList<>();
for(var row:targets){var ids=jdbc.query("SELECT id FROM source_candidate WHERE "+SourceIdentity.canonicalSlotSql("source_slot")+"=? AND sha256=? AND parser_config_hash=? AND deleted_at IS NULL AND status IN('DISCOVERED','IMPORTED') AND NOT "+SourceDeletionService.pendingDeletionSql("source_candidate")+" ORDER BY discovered_at DESC,id DESC LIMIT 1",(rs,n)->rs.getString(1),SourceIdentity.canonicalSlot((String)row.get("sourceSlot")),row.get("sha256"),row.get("parserConfigHash"));if(!ids.isEmpty())currentSourceIds.add(ids.getFirst());} for(var row:targets){var ids=jdbc.query("SELECT id FROM source_candidate WHERE "+SourceIdentity.canonicalSlotSql("source_slot")+"=? AND sha256=? AND parser_config_hash=? AND deleted_at IS NULL AND status IN('DISCOVERED','IMPORTED') AND NOT "+SourceDeletionService.pendingDeletionSql("source_candidate")+" ORDER BY discovered_at DESC,id DESC LIMIT 1",(rs,n)->rs.getString(1),SourceIdentity.canonicalSlot((String)row.get("sourceSlot")),row.get("sha256"),row.get("parserConfigHash"));if(!ids.isEmpty())currentSourceIds.add(ids.getFirst());}
@@ -41,39 +41,90 @@ public class SourceUploadService {
} }
@Transactional public Map<String,Object> configure(ConfigureRequest r,AppUserPrincipal original)throws IOException { @Transactional public Map<String,Object> configure(ConfigureRequest r,AppUserPrincipal original)throws IOException {
requireActor(original);AppUserPrincipal actor=access.resolvePrincipal(original); requireActor(original);AppUserPrincipal actor=access.resolvePrincipal(original);
if(r.year()<1900||r.year()>2100||r.month()<1||r.month()>12||!Set.of("ADD","REPLACE").contains(r.mode()==null?"":r.mode()))throw new IllegalArgumentException("请选择有效年月和新增/替换模式"); if(r.year()<1900||r.year()>2100||r.month()<1||r.month()>12||!Set.of("ADD","REPLACE","APPEND").contains(r.mode()==null?"":r.mode()))throw new IllegalArgumentException("请选择有效年月和新增、追加或替换模式");
boolean append="APPEND".equals(r.mode()),appendReplacement="REPLACE".equals(r.mode())&&r.fileId()!=null&&!r.fileId().isBlank();
if((append||appendReplacement)&&!validFileId(r.fileId()))throw new IllegalArgumentException("追加或替换追加文件时必须提供有效fileId");
if(!append&&!appendReplacement&&r.fileId()!=null&&!r.fileId().isBlank())throw new IllegalArgumentException("仅追加文件替换可提供fileId");
if(r.templateId()!=null&&!r.templateId().isBlank())throw new IllegalArgumentException("表头模板功能已停用,请直接提交工作表、表头行和字段映射"); if(r.templateId()!=null&&!r.templateId().isBlank())throw new IllegalArgumentException("表头模板功能已停用,请直接提交工作表、表头行和字段映射");
String reason=g.required(r.reason(),"操作理由");String scope=SourceIdentity.scope(r.scopeCode()==null?"ALL":r.scopeCode()); String reason=g.required(r.reason(),"操作理由");String scope=SourceIdentity.scope(r.scopeCode()==null?"ALL":r.scopeCode());
if(scope==null||scope.isBlank()||scope.length()>160||scope.contains("|")||!SourceIdentity.fullOrganizationScope(actor)&&(!actor.getDepartments().stream().map(SourceIdentity::scope).toList().contains(scope)||!actor.getGroups().isEmpty()&&!actor.getGroups().contains("*")))throw new ResponseStatusException(HttpStatus.FORBIDDEN,"来源范围须完整属于当前组织及分组授权"); if(scope==null||scope.isBlank()||scope.length()>160||scope.contains("|")||!SourceIdentity.fullOrganizationScope(actor)&&(!actor.getDepartments().stream().map(SourceIdentity::scope).toList().contains(scope)||!actor.getGroups().isEmpty()&&!actor.getGroups().contains("*")))throw new ResponseStatusException(HttpStatus.FORBIDDEN,"来源范围须完整属于当前组织及分组授权");
Map<String,Object> draft=draft(r.uploadId(),actor,true);if(draft.get("candidateId")!=null)throw new ResponseStatusException(HttpStatus.CONFLICT,"此上传已确认,请刷新来源列表或重新上传"); Map<String,Object> draft=draft(r.uploadId(),actor,true);String existingCandidateId=(String)draft.get("candidateId");Map<String,Object> priorCandidate=existingCandidateId==null?null:reconfigurableCandidate(existingCandidateId,actor,draft);
Path file=verified(draft);SourceParserConfig config=new SourceParserConfig(r.stage(),r.sheetName(),r.headerRow(),r.fieldMap(),null); Path file=verified(draft);SourceParserConfig config=new SourceParserConfig(r.stage(),r.sheetName(),r.headerRow(),r.fieldMap(),null);
Map<String,Object> actual=reader.preview(file,r.sheetName(),r.headerRow());Map<?,?> preview=(Map<?,?>)actual.get("preview");config.validate((List<String>)preview.get("headers")); Map<String,Object> actual=reader.preview(file,r.sheetName(),r.headerRow());Map<?,?> preview=(Map<?,?>)actual.get("preview");config.validate((List<String>)preview.get("headers"));
String current=current(true);if(!Objects.equals(current,r.expectedCurrentReleaseId()))throw new ResponseStatusException(HttpStatus.CONFLICT,"当前正式版本已变化,请刷新后重新确认"); String current=current(true);if(!Objects.equals(current,r.expectedCurrentReleaseId()))throw new ResponseStatusException(HttpStatus.CONFLICT,"当前正式版本已变化,请刷新后重新确认");
String slot=SourceIdentity.slot(r.stage(),r.year(),r.month(),scope,"MAIN"); String shard=(append||appendReplacement)?"UPLOAD_"+r.fileId():"MAIN";
String slot=SourceIdentity.slot(r.stage(),r.year(),r.month(),scope,shard);
if(slot.length()>190)throw new IllegalArgumentException("来源身份过长");
var documents=g.rows("SELECT id FROM source_document WHERE "+SourceIdentity.canonicalSlotSql("source_slot")+"=? FOR UPDATE",slot); var documents=g.rows("SELECT id FROM source_document WHERE "+SourceIdentity.canonicalSlotSql("source_slot")+"=? FOR UPDATE",slot);
String replacementRevision=null,replacementDocument=null; String replacementRevision=null,replacementDocument=null;
if("REPLACE".equals(r.mode())){ if("REPLACE".equals(r.mode())){
if(r.targetDocumentId()==null||r.targetRevisionId()==null||current==null)throw new IllegalArgumentException("请选择当前正式版中的替换来源"); if(r.targetDocumentId()==null||r.targetRevisionId()==null||current==null)throw new IllegalArgumentException("请选择当前正式版中的替换来源");
var old=g.rows("SELECT d.id,r.source_revision_id FROM release_member r JOIN source_document d ON d.id=r.source_document_id WHERE r.release_id=? AND d.id=? AND r.source_revision_id=? AND "+SourceIdentity.canonicalSlotSql("d.source_slot")+"=? FOR UPDATE",current,r.targetDocumentId(),r.targetRevisionId(),slot); var old=g.rows("SELECT d.id,r.source_revision_id,d.source_mode FROM release_member r JOIN source_document d ON d.id=r.source_document_id WHERE r.release_id=? AND d.id=? AND r.source_revision_id=? AND "+SourceIdentity.canonicalSlotSql("d.source_slot")+"=? FOR UPDATE",current,r.targetDocumentId(),r.targetRevisionId(),slot);
if(old.size()!=1)throw new ResponseStatusException(HttpStatus.CONFLICT,"替换目标的月份、阶段、范围或旧修订已变化");sourceAccess.requireSource(r.targetDocumentId(),r.targetRevisionId(),actor);replacementDocument=r.targetDocumentId();replacementRevision=r.targetRevisionId(); if(old.size()!=1)throw new ResponseStatusException(HttpStatus.CONFLICT,"替换目标的月份、阶段、范围、追加文件ID或旧修订已变化");
}else if(!documents.isEmpty())throw new ResponseStatusException(HttpStatus.CONFLICT,"该月份、阶段及范围已有来源,请选择替换旧数据"); if(appendReplacement&&!("APPEND_SNAPSHOT".equals(old.getFirst().get("sourceMode"))))throw new ResponseStatusException(HttpStatus.CONFLICT,"替换目标不是由追加上传创建的文件批次");
sourceAccess.requireSource(r.targetDocumentId(),r.targetRevisionId(),actor);replacementDocument=r.targetDocumentId();replacementRevision=r.targetRevisionId();
}else if(!documents.isEmpty())throw new ResponseStatusException(HttpStatus.CONFLICT,append?"该fileId已有来源,请选择这份追加文件并替换其当前修订":"该月份、阶段及范围已有来源,请选择替换旧数据");
if(append){
var prior=g.rows("SELECT id FROM source_candidate WHERE "+SourceIdentity.canonicalSlotSql("source_slot")+"=? AND deleted_at IS NULL AND status IN('DISCOVERED','QUEUED','RUNNING','IMPORTED') FOR UPDATE",slot);
if(prior.stream().anyMatch(row->!Objects.equals(existingCandidateId,row.get("id"))))throw new ResponseStatusException(HttpStatus.CONFLICT,"该fileId已登记来源,请选择已有来源替换,或为新的独立文件批次生成fileId");
}
Integer pending=jdbc.queryForObject("SELECT COUNT(*) FROM source_candidate WHERE "+SourceIdentity.canonicalSlotSql("source_slot")+"=? AND status IN('QUEUED','RUNNING')",Integer.class,slot);if(pending!=null&&pending>0)throw new ResponseStatusException(HttpStatus.CONFLICT,"该来源正在构建,请稍后重试"); Integer pending=jdbc.queryForObject("SELECT COUNT(*) FROM source_candidate WHERE "+SourceIdentity.canonicalSlotSql("source_slot")+"=? AND status IN('QUEUED','RUNNING')",Integer.class,slot);if(pending!=null&&pending>0)throw new ResponseStatusException(HttpStatus.CONFLICT,"该来源正在构建,请稍后重试");
String protocol=protocol(slot,r,scope,actor,reason);String key=GovernanceSupport.sha(slot+"\n"+draft.get("sha256")+"\n"+config.hash()+"\n"+r.uploadId());String id=UUID.randomUUID().toString(); ensureUniqueContent((String)draft.get("sha256"),r.stage(),r.year(),r.month(),scope,current,replacementDocument,existingCandidateId);
Map<String,Object> frozenConfiguration=new LinkedHashMap<>(config.snapshot());frozenConfiguration.put("sourceSlot",slot); String protocol=protocol(slot,r,scope,actor,reason,shard);String key=GovernanceSupport.sha(slot+"\n"+draft.get("sha256")+"\n"+config.hash()+"\n"+r.uploadId());String id=existingCandidateId==null?UUID.randomUUID().toString():existingCandidateId;
jdbc.update("INSERT INTO source_candidate(id,candidate_key,source_slot,stage,source_year,source_month,scope_code,source_file,source_path,sha256,file_size,status,protocol_id,resolution,resolution_reason,parser_config,parser_config_hash,expected_current_release_id,replacement_document_id,replacement_revision_id,configured_by) VALUES(?,?,?,?,?,?,?,?,?,?,?,'DISCOVERED',?,?,?,CAST(? AS JSON),?,?,?,?,?)",id,key,slot,r.stage(),r.year(),r.month(),scope,draft.get("sourceFile"),file.toString(),draft.get("sha256"),draft.get("fileSize"),protocol,"REPLACE".equals(r.mode())?"REPLACE":"CONFIRMED",reason,json.writeValueAsString(frozenConfiguration),config.hash(),current,replacementDocument,replacementRevision,actor.getId().toString()); Map<String,Object> frozenConfiguration=new LinkedHashMap<>(config.snapshot());frozenConfiguration.put("sourceSlot",slot);if(append||appendReplacement)frozenConfiguration.put("fileId",r.fileId());
jdbc.update("UPDATE source_upload_draft SET candidate_id=? WHERE id=?",id,r.uploadId());g.audit(actor.getId().toString(),"SOURCE_UPLOAD_CONFIGURED","source_candidate",id,Map.of("mode",r.mode(),"slot",slot,"reason",reason,"configuration",frozenConfiguration)); String resolution=append?"APPEND_CONFIRMED":"REPLACE".equals(r.mode())?"REPLACE":"CONFIRMED";
Map<String,Object> result=new LinkedHashMap<>();result.put("source",new SourceCatalogService.SourceView(id,(String)draft.get("sourceFile"),r.stage(),r.year(),r.month(),((Number)draft.get("fileSize")).longValue(),(String)draft.get("sha256"),"DISCOVERED"));result.put("configuration",frozenConfiguration);result.put("currentReleaseId",current);return result; if(priorCandidate==null){
jdbc.update("INSERT INTO source_candidate(id,candidate_key,source_slot,stage,source_year,source_month,scope_code,source_file,source_path,sha256,file_size,status,protocol_id,resolution,resolution_reason,parser_config,parser_config_hash,expected_current_release_id,replacement_document_id,replacement_revision_id,configured_by) VALUES(?,?,?,?,?,?,?,?,?,?,?,'DISCOVERED',?,?,?,CAST(? AS JSON),?,?,?,?,?)",id,key,slot,r.stage(),r.year(),r.month(),scope,draft.get("sourceFile"),file.toString(),draft.get("sha256"),draft.get("fileSize"),protocol,resolution,reason,json.writeValueAsString(frozenConfiguration),config.hash(),current,replacementDocument,replacementRevision,actor.getId().toString());
}else{
String oldSlot=SourceIdentity.canonicalSlot((String)priorCandidate.get("sourceSlot"));String oldProtocol=Objects.toString(priorCandidate.get("protocolId"),null);
if(!Objects.equals(oldSlot,SourceIdentity.canonicalSlot(slot))&&oldProtocol!=null)deactivateUnusedUploadProtocol(oldProtocol,id);
int changed=jdbc.update("UPDATE source_candidate SET candidate_key=?,source_slot=?,stage=?,source_year=?,source_month=?,scope_code=?,source_file=?,source_path=?,sha256=?,file_size=?,protocol_id=?,resolution=?,resolution_reason=?,parser_config=CAST(? AS JSON),parser_config_hash=?,expected_current_release_id=?,replacement_document_id=?,replacement_revision_id=?,configured_by=? WHERE id=? AND status='DISCOVERED' AND deleted_at IS NULL",key,slot,r.stage(),r.year(),r.month(),scope,draft.get("sourceFile"),file.toString(),draft.get("sha256"),draft.get("fileSize"),protocol,resolution,reason,json.writeValueAsString(frozenConfiguration),config.hash(),current,replacementDocument,replacementRevision,actor.getId().toString(),id);
if(changed!=1)throw new ResponseStatusException(HttpStatus.CONFLICT,"来源已排队或状态变化,请刷新来源列表后重试");
}
jdbc.update("UPDATE source_upload_draft SET candidate_id=? WHERE id=?",id,r.uploadId());g.audit(actor.getId().toString(),priorCandidate==null?"SOURCE_UPLOAD_CONFIGURED":"SOURCE_UPLOAD_RECONFIGURED","source_candidate",id,Map.of("mode",r.mode(),"slot",slot,"reason",reason,"configuration",frozenConfiguration));
Map<String,Object> result=new LinkedHashMap<>();result.put("source",new SourceCatalogService.SourceView(id,(String)draft.get("sourceFile"),r.stage(),r.year(),r.month(),((Number)draft.get("fileSize")).longValue(),(String)draft.get("sha256"),"DISCOVERED"));result.put("configuration",frozenConfiguration);result.put("currentReleaseId",current);result.put("reconfigured",priorCandidate!=null);return result;
} }
private String protocol(String slot,ConfigureRequest r,String scope,AppUserPrincipal actor,String reason){ private String protocol(String slot,ConfigureRequest r,String scope,AppUserPrincipal actor,String reason,String shard){
String mode="MAIN".equals(shard)?"MONTHLY_SNAPSHOT":"APPEND_SNAPSHOT";
var existing=g.rows("SELECT id,source_mode,applicable FROM source_protocol WHERE "+SourceIdentity.canonicalSlotSql("source_slot")+"=? AND active=TRUE FOR UPDATE",slot); var existing=g.rows("SELECT id,source_mode,applicable FROM source_protocol WHERE "+SourceIdentity.canonicalSlotSql("source_slot")+"=? AND active=TRUE FOR UPDATE",slot);
if(existing.size()>1)throw new IllegalArgumentException("来源协议存在重复,请先完成来源治理"); if(existing.size()>1)throw new IllegalArgumentException("来源协议存在重复,请先完成来源治理");
if(!existing.isEmpty()){if(!"MONTHLY_SNAPSHOT".equals(existing.getFirst().get("sourceMode"))||!Boolean.TRUE.equals(existing.getFirst().get("applicable")))throw new IllegalArgumentException("此向导仅接收适用阶段的完整月度快照");return (String)existing.getFirst().get("id");} if(!existing.isEmpty()){if(!mode.equals(existing.getFirst().get("sourceMode"))||!Boolean.TRUE.equals(existing.getFirst().get("applicable")))throw new IllegalArgumentException("来源协议与完整快照/追加批次身份不匹配,须先完成来源治理");return (String)existing.getFirst().get("id");}
String id=UUID.randomUUID().toString();Integer version=jdbc.queryForObject("SELECT COALESCE(MAX(version_no),0)+1 FROM source_protocol WHERE "+SourceIdentity.canonicalSlotSql("source_slot")+"=?",Integer.class,slot);jdbc.update("INSERT INTO source_protocol(id,source_slot,version_no,stage,source_year,source_month,scope_code,source_mode,required_sheet,applicable,expected,reason,created_by) VALUES(?,?,?,?,?,?,?,'MONTHLY_SNAPSHOT',?,TRUE,TRUE,?,?)",id,slot,version,r.stage(),r.year(),r.month(),scope,r.sheetName(),reason,actor.getId().toString());return id; var versions=g.rows("SELECT version_no FROM source_protocol WHERE "+SourceIdentity.canonicalSlotSql("source_slot")+"=? ORDER BY version_no DESC LIMIT 1 FOR UPDATE",slot);int version=versions.isEmpty()?1:((Number)versions.getFirst().get("versionNo")).intValue()+1;
String id=UUID.randomUUID().toString();jdbc.update("INSERT INTO source_protocol(id,source_slot,version_no,stage,source_year,source_month,scope_code,shard_code,source_mode,required_sheet,applicable,expected,reason,created_by) VALUES(?,?,?,?,?,?,?,?,?,?,TRUE,TRUE,?,?)",id,slot,version,r.stage(),r.year(),r.month(),scope,shard,mode,r.sheetName(),reason,actor.getId().toString());return id;
} }
private Map<String,Object> reconfigurableCandidate(String id,AppUserPrincipal actor,Map<String,Object> draft){
var rows=g.rows("SELECT * FROM source_candidate WHERE id=? AND configured_by=? FOR UPDATE",id,actor.getId().toString());
if(rows.isEmpty())throw new ResponseStatusException(HttpStatus.CONFLICT,"已确认来源不属于当前上传人,请重新上传");Map<String,Object> candidate=rows.getFirst();
if(!"DISCOVERED".equals(candidate.get("status"))||!Objects.equals(candidate.get("sha256"),draft.get("sha256"))||!Objects.equals(candidate.get("sourcePath"),draft.get("sourcePath")))throw new ResponseStatusException(HttpStatus.CONFLICT,"此来源已排队、处理或内容变化,不能重新确认");
Integer runs=jdbc.queryForObject("SELECT COUNT(*) FROM import_run WHERE JSON_CONTAINS(requested_sources,JSON_QUOTE(?))",Integer.class,id);
String sameSlot=SourceIdentity.canonicalSlotSql("d.source_slot")+"="+SourceIdentity.canonicalSlotSql("c.source_slot");
Integer documents=jdbc.queryForObject("SELECT COUNT(*) FROM source_candidate c JOIN source_document d ON "+sameSlot+" JOIN source_revision r ON r.source_document_id=d.id AND r.sha256=c.sha256 AND r.parser_config_hash=c.parser_config_hash JOIN release_member m ON m.source_document_id=d.id AND m.source_revision_id=r.id WHERE c.id=?",Integer.class,id);
if((runs!=null&&runs>0)||(documents!=null&&documents>0))throw new ResponseStatusException(HttpStatus.CONFLICT,"来源已被导入任务或版本引用,不能重新确认");
return candidate;
}
private void deactivateUnusedUploadProtocol(String protocolId,String candidateId){
var protocol=g.rows("SELECT source_slot,active FROM source_protocol WHERE id=? AND active=TRUE FOR UPDATE",protocolId);if(protocol.isEmpty())return;
Integer others=jdbc.queryForObject("SELECT COUNT(*) FROM source_candidate WHERE protocol_id=? AND id<>? AND deleted_at IS NULL AND status NOT IN('RETURNED','SUPERSEDED')",Integer.class,protocolId,candidateId);
Integer documents=jdbc.queryForObject("SELECT COUNT(*) FROM source_document WHERE "+SourceIdentity.canonicalSlotSql("source_slot")+"=?",Integer.class,SourceIdentity.canonicalSlot((String)protocol.getFirst().get("sourceSlot")));
if((others==null||others==0)&&(documents==null||documents==0))jdbc.update("UPDATE source_protocol SET active=FALSE WHERE id=? AND active=TRUE",protocolId);
}
private void ensureUniqueContent(String sha,String stage,int year,int month,String scope,String releaseId,String excludedDocument,String excludedCandidate){
if(releaseId!=null){
List<Map<String,Object>> currentRows=g.rows("SELECT d.id,d.scope_code FROM release_member m JOIN source_document d ON d.id=m.source_document_id JOIN source_revision r ON r.id=m.source_revision_id WHERE m.release_id=? AND r.sha256=? AND d.stage=? AND d.source_year=? AND d.source_month=? FOR UPDATE",releaseId,sha,stage,year,month);
if(currentRows.stream().anyMatch(row->!Objects.equals(excludedDocument,row.get("id"))&&SourceIdentity.intersects(scope,(String)row.get("scopeCode"))))throw duplicateContent();
}
List<Map<String,Object>> pending=g.rows("SELECT id,source_slot,scope_code FROM source_candidate WHERE sha256=? AND stage=? AND source_year=? AND source_month=? AND deleted_at IS NULL AND status IN('DISCOVERED','QUEUED','RUNNING') AND NOT "+SourceDeletionService.pendingDeletionSql("source_candidate")+" FOR UPDATE",sha,stage,year,month);
if(pending.stream().anyMatch(row->!Objects.equals(excludedCandidate,row.get("id"))&&SourceIdentity.intersects(scope,(String)row.get("scopeCode"))))throw duplicateContent();
}
private ResponseStatusException duplicateContent(){return new ResponseStatusException(HttpStatus.CONFLICT,"该工作簿内容已用于同一年月、阶段及重叠范围的另一份来源,请勿重复计入;更正版请精确选择原来源替换");}
private static boolean validFileId(String value){return value!=null&&value.length()<=80&&value.matches("[A-Za-z0-9_-]{1,80}");}
private static String fileId(Object slotValue){if(!(slotValue instanceof String slot))return null;String[] parts=SourceIdentity.canonicalSlot(slot).split("\\|",-1);if(parts.length!=5||!parts[4].startsWith("UPLOAD_"))return null;String value=parts[4].substring("UPLOAD_".length());return validFileId(value)?value:null;}
private Map<String,Object> draft(String id,AppUserPrincipal actor,boolean lock){var rows=g.rows("SELECT * FROM source_upload_draft WHERE id=? AND created_by=? AND expires_at>CURRENT_TIMESTAMP(6)"+(lock?" FOR UPDATE":""),id,actor.getId().toString());if(rows.isEmpty())throw new ResponseStatusException(HttpStatus.NOT_FOUND,"上传预览已过期或不属于当前用户,请重新上传");return rows.getFirst();} private Map<String,Object> draft(String id,AppUserPrincipal actor,boolean lock){var rows=g.rows("SELECT * FROM source_upload_draft WHERE id=? AND created_by=? AND expires_at>CURRENT_TIMESTAMP(6)"+(lock?" FOR UPDATE":""),id,actor.getId().toString());if(rows.isEmpty())throw new ResponseStatusException(HttpStatus.NOT_FOUND,"上传预览已过期或不属于当前用户,请重新上传");return rows.getFirst();}
private Path verified(Map<String,Object> draft)throws IOException{Path path=Path.of((String)draft.get("sourcePath"));if(Files.isSymbolicLink(path)||!Files.isRegularFile(path)||!path.toRealPath().startsWith(scanner.archiveRoot())||Files.size(path)!=((Number)draft.get("fileSize")).longValue()||!SourceScanner.sha256(path).equals(draft.get("sha256")))throw new IllegalArgumentException("上传文件已变化,请重新上传");return path.toRealPath();} private Path verified(Map<String,Object> draft)throws IOException{Path path=Path.of((String)draft.get("sourcePath"));if(Files.isSymbolicLink(path)||!Files.isRegularFile(path)||!path.toRealPath().startsWith(scanner.archiveRoot())||Files.size(path)!=((Number)draft.get("fileSize")).longValue()||!SourceScanner.sha256(path).equals(draft.get("sha256")))throw new IllegalArgumentException("上传文件已变化,请重新上传");return path.toRealPath();}
private String current(boolean lock){return jdbc.queryForObject("SELECT release_id FROM current_release WHERE dataset_id='quality'"+(lock?" FOR UPDATE":""),String.class);} private String current(boolean lock){return jdbc.queryForObject("SELECT release_id FROM current_release WHERE dataset_id='quality'"+(lock?" FOR UPDATE":""),String.class);}
private void requireActor(AppUserPrincipal actor){access.requireUserActivePolicy(actor);access.requireBoundAction("RECEIVER","source.receive");} private void requireActor(AppUserPrincipal actor){access.requireUserActivePolicy(actor);access.requireBoundAction("RECEIVER","source.receive");}
/** templateId remains an explicitly disabled compatibility field for older clients. */ /** templateId remains an explicitly disabled compatibility field for older clients. */
public record ConfigureRequest(String uploadId,String stage,int year,int month,String sheetName,int headerRow,Map<String,String> fieldMap,String templateId,String mode,String expectedCurrentReleaseId,String targetDocumentId,String targetRevisionId,String reason,String scopeCode){} public record ConfigureRequest(String uploadId,String stage,int year,int month,String sheetName,int headerRow,Map<String,String> fieldMap,String templateId,String mode,String expectedCurrentReleaseId,String targetDocumentId,String targetRevisionId,String reason,String scopeCode,String fileId){
public ConfigureRequest(String uploadId,String stage,int year,int month,String sheetName,int headerRow,Map<String,String> fieldMap,String templateId,String mode,String expectedCurrentReleaseId,String targetDocumentId,String targetRevisionId,String reason,String scopeCode){this(uploadId,stage,year,month,sheetName,headerRow,fieldMap,templateId,mode,expectedCurrentReleaseId,targetDocumentId,targetRevisionId,reason,scopeCode,null);}
}
} }
@@ -0,0 +1,27 @@
package cn.yunsheng.quality.update;
import java.util.Map;
import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequestMapping("/api/system-update")
@PreAuthorize("hasRole('SYSTEM_ADMIN')")
public class SystemUpdateController {
private final SystemUpdateService updates;
public SystemUpdateController(SystemUpdateService updates) { this.updates = updates; }
@GetMapping
public Map<String, Object> status() { return updates.status(); }
// Both mutations use the application's existing session + CSRF filter.
@PostMapping("/check")
public Map<String, Object> check() { return updates.check(); }
@PostMapping("/apply")
public Map<String, Object> apply() { return updates.apply(); }
}
@@ -0,0 +1,39 @@
package cn.yunsheng.quality.update;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.Set;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;
/** Keeps the startup-to-update-commit interval free of business writes. */
@Component
@Order(Ordered.HIGHEST_PRECEDENCE + 20)
public class SystemUpdateMaintenanceFilter extends OncePerRequestFilter {
private static final Set<String> READ_METHODS = Set.of("GET", "HEAD", "OPTIONS", "TRACE");
private static final Set<String> AUTH_ROUTES = Set.of("/api/auth/login", "/api/auth/logout", "/api/auth/csrf", "/api/auth/me");
private final SystemUpdateService updates;
public SystemUpdateMaintenanceFilter(SystemUpdateService updates) { this.updates = updates; }
@Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain)
throws ServletException, IOException {
String path = request.getRequestURI().substring(request.getContextPath().length());
if (!READ_METHODS.contains(request.getMethod()) && path.startsWith("/api/")
&& !path.equals("/api/system-update") && !path.startsWith("/api/system-update/")
&& !AUTH_ROUTES.contains(path) && updates.maintenanceActive()) {
response.setStatus(HttpServletResponse.SC_SERVICE_UNAVAILABLE);
response.setContentType("application/problem+json;charset=UTF-8");
response.setHeader("Retry-After", "10");
response.getWriter().write("{\"title\":\"Service Unavailable\",\"status\":503,\"detail\":\"系统正在更新,请稍后重试。\"}");
return;
}
chain.doFilter(request, response);
}
}
@@ -0,0 +1,142 @@
package cn.yunsheng.quality.update;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Service;
import org.springframework.web.server.ResponseStatusException;
import tools.jackson.databind.json.JsonMapper;
/** Restricted bridge to the packaged updater; neither URLs nor paths are input. */
@Service
public class SystemUpdateService {
private static final Set<String> RESPONSE_FIELDS = Set.of("enabled", "currentVersion", "state", "message",
"checkedAt", "version", "available", "startedAt", "finishedAt");
private final JsonMapper json;
private final Path root;
private final boolean enabled;
public SystemUpdateService(JsonMapper json,
@Value("${QUALITY_PROJECT_ROOT:}") String root,
@Value("${QUALITY_UPDATE_ENABLED:false}") boolean requested) {
this.json = json;
this.root = root.isBlank() ? null : Path.of(root).toAbsolutePath().normalize();
this.enabled = requested && System.getProperty("os.name", "").startsWith("Windows")
&& this.root != null && Files.isRegularFile(this.root.resolve("manifest.json"))
&& Files.isRegularFile(this.root.resolve("runtimes/python/python.exe"))
&& Files.isRegularFile(this.root.resolve("deploy/offline/update_manager.py"))
&& Files.isRegularFile(this.root.resolve("deploy/offline/update-public-key.pem"));
}
public synchronized Map<String, Object> status() {
if (!enabled) return Map.of("enabled", false, "state", "disabled", "message", "此部署不支持软件内更新。");
return invoke("status");
}
public synchronized Map<String, Object> check() {
requireEnabled();
return invoke("check");
}
public synchronized Map<String, Object> apply() {
requireEnabled();
return invoke("launch");
}
/** Fail closed during installation/recovery, including a process crash. */
public boolean maintenanceActive() {
if (!enabled) return false;
try {
Map<?, ?> state = readMetadata("status.json");
Map<?, ?> journal = readMetadata("journal.json");
// 同一任务的持久提交记录优先于尚未成功刷新的进度提示。
if (journal.get("operationId") instanceof String operationId && !operationId.isBlank()
&& operationId.equals(state.get("operationId"))
&& Set.of("complete", "rolledBack").contains(journal.get("phase"))) return false;
if (state.get("state") instanceof String phase
&& Set.of("backingUp", "installing", "restarting", "rollingBack").contains(phase)) return true;
return journal.get("phase") instanceof String phase && Set.of("stopping", "installing").contains(phase);
} catch (Exception failure) {
return true;
}
}
private Map<?, ?> readMetadata(String name) throws IOException {
Path path = root.resolve(".runtime/update/" + name);
if (!Files.exists(path)) return Map.of();
try (var stream = Files.newInputStream(path)) {
byte[] raw = stream.readNBytes(8193);
if (raw.length > 8192) throw new IOException("Oversized update state");
Object decoded = json.readValue(raw, Object.class);
if (!(decoded instanceof Map<?, ?> value)) throw new IOException("Invalid update state");
return value;
}
}
private void requireEnabled() {
if (!enabled) throw new ResponseStatusException(HttpStatus.CONFLICT, "此部署不支持软件内更新。");
}
private Map<String, Object> invoke(String action) {
Process process = null;
try {
ProcessBuilder builder = new ProcessBuilder(root.resolve("runtimes/python/python.exe").toString(),
root.resolve("deploy/offline/update_manager.py").toString(), action);
builder.directory(root.toFile());
builder.redirectError(ProcessBuilder.Redirect.DISCARD);
builder.environment().put("PYTHONUTF8", "1");
builder.environment().put("PYTHONIOENCODING", "utf-8");
builder.environment().put("PYTHONDONTWRITEBYTECODE", "1");
process = builder.start();
process.getOutputStream().close();
// The CLI prints one small, allowlisted JSON receipt. Waiting first
// also bounds hangs caused by a failed DNS/TLS/network operation.
if (!process.waitFor(45, TimeUnit.SECONDS)) {
process.destroyForcibly();
throw new ResponseStatusException(HttpStatus.GATEWAY_TIMEOUT, "更新服务器暂未响应,请稍后重试。");
}
byte[] raw = process.getInputStream().readNBytes(8193);
if (raw.length > 8192) throw new IOException("Oversized updater receipt");
Object decoded = json.readValue(new String(raw, StandardCharsets.UTF_8), Object.class);
if (!(decoded instanceof Map<?, ?> value)) throw new IOException("Invalid updater receipt");
if (process.exitValue() != 0) {
Object detail = value.get("error");
String message = detail instanceof String s && s.length() <= 300 ? s : "更新操作未完成,请稍后重试。";
throw new ResponseStatusException(HttpStatus.CONFLICT, message);
}
return safeResponse(value);
} catch (ResponseStatusException failure) {
throw failure;
} catch (InterruptedException failure) {
Thread.currentThread().interrupt();
throw new ResponseStatusException(HttpStatus.SERVICE_UNAVAILABLE, "更新操作已中断。");
} catch (Exception failure) {
throw new ResponseStatusException(HttpStatus.SERVICE_UNAVAILABLE, "更新服务暂不可用,请稍后重试。");
} finally {
if (process != null && process.isAlive()) process.destroyForcibly();
}
}
static Map<String, Object> safeResponse(Map<?, ?> value) throws IOException {
Map<String, Object> result = new LinkedHashMap<>();
for (String field : RESPONSE_FIELDS) {
Object item = value.get(field);
if (item == null) continue;
if ((field.equals("enabled") || field.equals("available")) && !(item instanceof Boolean))
throw new IOException("Invalid updater boolean");
if (!(item instanceof Boolean) && !(item instanceof String s && s.length() <= 300))
throw new IOException("Invalid updater field");
result.put(field, item);
}
if (!(result.get("state") instanceof String) || !(result.get("enabled") instanceof Boolean))
throw new IOException("Incomplete updater receipt");
return result;
}
}

Some files were not shown because too many files have changed in this diff Show More