完善:同月多文件导入与当前用户离线迁移更新

This commit is contained in:
2026-11-04 11:35:24 +08:00
parent 147b56d0fd
commit f04d4e96e3
89 changed files with 6634 additions and 201 deletions
+36
View File
@@ -0,0 +1,36 @@
# 质量中心离线部署
此目录是 Windows / Linux 原生离线包的打包与启动工具。编译后的前端、Spring Boot fat JAR、Java 21、MySQL 8.4、Python 3.11.15 和固定版本的 AI 依赖全部随包交付;目标机不执行 pip、pnpm、Maven 或联网下载。不需要 Docker。
Windows 推荐将离线包解压到 `%LOCALAPPDATA%\QualityCenter`,或本机固定 NTFS 磁盘上更短的当前用户可写目录。启动器会读取包内 `manifest.json`,要求部署根路径长度加最长相对文件路径不超过 248 个字符;若用户名较长导致 `%LOCALAPPDATA%\QualityCenter` 超限,请选择更短的用户可写目录。请用当前用户运行 `windows-start.ps1`;服务进程在当前用户会话中启动,运行环境随包提供,不安装系统组件、不注册全局服务,也不修改 Windows 防火墙。网页服务会监听内网网卡并显示本机及内网访问链接;MySQL 和后端接口只监听 `127.0.0.1`。其他设备能否访问网页取决于系统现有防火墙和网络策略,需要放行时由用户或管理员按组织策略处理。
Linux 默认只允许本机访问,使用 `./start.sh --lan` 开启内网监听;已有防火墙策略由部署者按实际网段配置。使用普通用户启动,不能以 root 运行 MySQL。
`build_bundle.py --work output/offline-20261104` 使用隔离构建目录中的 JAR / 前端产物及已下载的官方发行文件组装包。`downloads/assets.json` 记录来源及摘要,`requirements-*.lock` 固定各平台依赖,`wheels-*` 是已通过 pip 哈希验证的平台 wheel。该命令不会读取本机数据库、用户配置、API 密钥或 AI 会话。
为保留本机运行环境,跨平台适配只应用在包内的 Python 脚本副本上;原有 macOS 脚本不被改写。适配包括 Windows 的文件锁、进程创建时间校验和 NTFS 继承权限,以及随包 Python / nao-core 的相对位置。
包内 `manifest.json` 记录每个只读文件的 SHA-256 和符号链接目标。首次及后续启动都会检查文件完整性和平台原生程序;运行数据单独保存在 `.runtime/` 和 `runtimedata/`。初始管理员密码在目标机随机生成,不使用公共默认密码。
包内不包含业务数据库、工作簿、现有用户、模型密钥或本机私有配置。全离线时可使用常规查询、导入和分析功能;AI 调用在线模型时需要网络,或自行配置可达的内网模型服务。代码执行沙箱和开发测试工具不在业务部署范围内。
不能在服务运行时移动安装目录。迁移业务数据应使用逻辑备份与对应版本的恢复流程,不得直接将 macOS 的 MySQL 数据目录复制到其他系统。
## Windows 增量更新包(维护人员)
离线签名工具读取 Windows x64 部署包的 `manifest.json`,只打包 `update_manager.program_path` 认可的完整程序清单,并使用包内运行环境和 wheel 信息计算运行时指纹。私有配置、业务数据、迁移文件、启动引导和密钥不会进入更新包。普通发布的序号必须大于部署包当前的 `updateSequence`。首次通道种子可显式加 `--allow-baseline`,但版本和序号必须分别与 `updateVersion`、`updateSequence` 完全相同;低序号始终拒绝。私钥必须是与包内更新公钥匹配的 Ed25519 PEM 文件,并保存在部署包及产物目录之外。
```sh
python3 deploy/offline/publish_update.py \
--root /path/to/QualityCenter-Windows-x64 \
--version 2026.11.04.2 \
--sequence 2 \
--private-key /secure/path/update-private-key.pem \
--output-dir /path/to/new-update-output
```
若需要为 `updateVersion=2026.11.04.1`、`updateSequence=1` 的原始包签署首个通道基线,使用与该包完全相同的版本和序号,并显式传入 `--allow-baseline`。不要修改包内 manifest 来绕过检查。
工具只在本地产生 `quality-update-<version>.zip` 和签名 `latest.json`,不会上传或发布文件。每个版本请使用新的输出目录;发布到更新服务器由维护人员另行完成。
参考:[Java 21 官方发行](https://adoptium.net/temurin/releases?version=21)、[MySQL Windows 安装要求](https://dev.mysql.com/doc/refman/8.4/en/windows-installation.html)、[MySQL Linux 二进制安装](https://dev.mysql.com/doc/refman/8.4/en/binary-installation.html)、[Python standalone](https://github.com/astral-sh/python-build-standalone)、[nao-core 0.3.26](https://pypi.org/project/nao-core/0.3.26/)、[Microsoft 运行库](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist/)。
+330
View File
@@ -0,0 +1,330 @@
#!/usr/bin/env python3
"""Assemble native offline packages from reviewed builds and verified downloads."""
from __future__ import annotations
import argparse
from datetime import datetime, timezone
import hashlib
import io
import json
from pathlib import Path
import re
import shutil
import stat
import subprocess
import tarfile
import tempfile
import zipfile
ROOT = Path(__file__).resolve().parents[2]
TEMPLATES = Path(__file__).resolve().parent
def digest(path, algorithm='sha256'):
with path.open('rb') as stream:
return hashlib.file_digest(stream, algorithm).hexdigest()
def extract(archive, target):
target.mkdir(parents=True)
with tempfile.TemporaryDirectory(dir=target.parent) as directory:
stage = Path(directory)
if archive.suffix == '.zip':
with zipfile.ZipFile(archive) as source:
source.extractall(stage)
else:
with tarfile.open(archive) as source:
# Terminfo contains case-distinct names. They alias on a Mac's
# default filesystem and are unused by this noninteractive service.
members = [member for member in source.getmembers()
if '/share/terminfo/' not in member.name and not member.name.endswith('/share/terminfo')]
source.extractall(stage, members=members, filter='data')
roots = list(stage.iterdir())
if len(roots) != 1 or not roots[0].is_dir():
raise RuntimeError('Expected one archive root: ' + archive.name)
shutil.copytree(roots[0], target, symlinks=True, dirs_exist_ok=True)
def extract_deb(archive, target):
raw = archive.read_bytes()
if raw[:8] != b'!<arch>\n':
raise RuntimeError('Invalid Debian archive')
position = 8
while position + 60 <= len(raw):
header = raw[position:position + 60]
name = header[:16].decode().strip().rstrip('/')
length = int(header[48:58].decode().strip())
position += 60
data = raw[position:position + length]
position += length + length % 2
if name.startswith('data.tar'):
with tempfile.TemporaryDirectory(dir=target.parent) as directory:
if name.endswith('.zst'):
import zstandard
with zstandard.ZstdDecompressor().stream_reader(io.BytesIO(data)) as reader:
with tarfile.open(fileobj=reader, mode='r|') as source:
source.extractall(directory, filter='data')
else:
with tarfile.open(fileobj=io.BytesIO(data)) as source:
source.extractall(directory, filter='data')
for path in Path(directory).rglob('*'):
if '.so' in path.name and (path.is_file() or path.is_symlink()):
dest = target / path.name
if path.is_symlink():
dest.symlink_to(Path(path.readlink()).name)
else:
shutil.copy2(path, dest)
docs = Path(directory) / 'usr/share/doc'
if docs.exists():
shutil.copytree(docs, target / 'licenses', dirs_exist_ok=True, symlinks=True)
return
raise RuntimeError('No data member in ' + archive.name)
def install_wheel(wheel, python_root, windows):
site = python_root / ('Lib/site-packages' if windows else 'lib/python3.11/site-packages')
site.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile(wheel) as source:
for member in source.infolist():
if member.is_dir():
continue
relative = Path(member.filename)
if relative.is_absolute() or '..' in relative.parts:
raise RuntimeError('Wheel path escape')
target = site / relative
if relative.parts[0].endswith('.data'):
scheme = relative.parts[1]
prefix = {'purelib': site, 'platlib': site, 'data': python_root,
'scripts': python_root / ('Scripts' if windows else 'bin'),
'headers': python_root / 'include'}[scheme]
target = prefix.joinpath(*relative.parts[2:])
target.parent.mkdir(parents=True, exist_ok=True)
with source.open(member) as reader, target.open('wb') as writer:
shutil.copyfileobj(reader, writer)
mode = member.external_attr >> 16
target.chmod((mode & 0o777) or 0o644)
def adapt_runtime(source):
def replace(old, new, count=None):
nonlocal source
actual = source.count(old)
if not actual or (count is not None and actual != count):
raise RuntimeError('Runtime source changed; inspect the portability patch: ' + old)
source = source.replace(old, new)
replace('import fcntl\n', 'import runtime_platform as _host\nfrom runtime_platform import fcntl\n', 1)
replace('PYTHON = VENV / "bin" / "python"',
'PYTHON = Path(os.environ.get("QUALITY_ASSISTANT_PYTHON", str(VENV / "bin" / "python")))', 1)
old = 'VENV / "lib" / "python3.11" / "site-packages" / "nao_core"'
replace(old, 'Path(os.environ.get("QUALITY_ASSISTANT_PACKAGE", str(' + old + ')))', 2)
replace('/ "nao-chat-server"', '/ ("nao-chat-server.exe" if os.name == "nt" else "nao-chat-server")', 1)
source, count = re.subn(r'\b(info|directory_info)\.st_mode & 0o077',
lambda match: '_host.private_bits(' + match[1] + '.st_mode)', source)
if count != 3:
raise RuntimeError('Runtime private-permission checks changed; inspect before packaging')
replace('os.fchmod(', '_host.fchmod(')
replace('os.getpgid(pid) != pid', 'not _host.group_leader(pid)', 1)
replace('os.killpg(', '_host.killpg(')
replace('signal.SIGKILL', 'getattr(signal, "SIGKILL", 9)')
replace('def _ps_value(pid: int, field: str) -> str | None:\n',
'def _ps_value(pid: int, field: str) -> str | None:\n'
' if os.name == "nt":\n return _host.ps_value(pid, field)\n', 1)
compile(source, 'nao_quality_runtime.py', 'exec')
return source
def adapt_script(name, source):
if name == 'nao_quality_runtime.py':
return adapt_runtime(source)
if name == 'analysis_visual_evidence.py':
source = source.replace('import stat\n', 'import stat\nimport runtime_platform as _host\n', 1)
source = source.replace('os.O_NOFOLLOW', 'getattr(os, "O_NOFOLLOW", 0)')
source = source.replace('os.O_NONBLOCK', 'getattr(os, "O_NONBLOCK", 0)')
source = source.replace('before.st_mode & 0o077', '_host.private_bits(before.st_mode)')
source = source.replace('before.st_uid != os.getuid()', '_host.owner_mismatch(before)')
source = source.replace(' fd = os.open(path,',
' if path.is_symlink():\n raise ValueError("invalid evidence link")\n fd = os.open(path,', 1)
elif name == 'nao_evidence_finalizer.py':
source = source.replace('import copy\n', 'import copy\nimport runtime_platform as _host\n', 1)
source = source.replace('path.stat().st_mode & 0o077', '_host.private_bits(path.stat().st_mode)')
compile(source, name, 'exec')
return source
def windows_local_runtime(root):
"""Use the signed vendor JRE's redistributable runtime; never install globally."""
libraries = ('msvcp140.dll', 'vcruntime140.dll', 'vcruntime140_1.dll')
for name in libraries:
source = root / 'runtimes/java/bin' / name
if not source.is_file():
raise RuntimeError('The official JRE is missing an app-local Microsoft runtime: ' + name)
for folder in ('runtimes/mysql/bin', 'runtimes/mysql/lib'):
shutil.copy2(source, root / folder / name)
mysql = root / 'runtimes/mysql'
# Debug-only tools depend on non-redistributable Microsoft debug runtimes.
for path in sorted(mysql.rglob('*'), reverse=True):
if path.is_file() and (any(part.lower() == 'debug' for part in path.relative_to(mysql).parts)
or '-debug.' in path.name.lower()):
path.unlink()
elif path.is_dir() and path.name.lower() == 'debug':
shutil.rmtree(path)
return {'name': 'app-local-msvc', 'os': 'windows', 'source': 'bundled-official-temurin-jre',
'libraries': {name: digest(root / 'runtimes/java/bin' / name) for name in libraries}}
def payload(root, work, osname, assets):
windows = osname == 'windows'
app = root / 'app'
app.mkdir(parents=True)
jars = list((work / 'source/backend/target').glob('*.jar'))
if len(jars) != 1:
raise RuntimeError('Expected one freshly built backend JAR')
with zipfile.ZipFile(jars[0]) as archive:
if 'BOOT-INF/classes/verification/original-workbook-acceptance.json' in archive.namelist():
raise RuntimeError('Exclude the private workbook acceptance input before building a distributable JAR')
shutil.copy2(jars[0], app / 'backend.jar')
shutil.copytree(work / 'source/web/dist', app / 'web')
for name in ('java', 'mysql', 'python'):
kind = 'jre' if name == 'java' else name
item = next(a for a in assets if a['name'] == kind and a['os'] == osname)
archive = work / 'downloads' / item['file']
if digest(archive) != item['sha256']:
raise RuntimeError('Download checksum mismatch: ' + item['file'])
extract(archive, root / 'runtimes' / name)
wheels = sorted((work / ('wheels-' + osname)).glob('*.whl'))
if len(wheels) < 60:
raise RuntimeError('The offline Python dependency set is incomplete')
lock_text = (work / f'requirements-{osname}.lock').read_text()
for wheel in wheels:
if '--hash=sha256:' + digest(wheel) not in lock_text:
raise RuntimeError('Wheel absent from hash lock: ' + wheel.name)
install_wheel(wheel, root / 'runtimes/python', windows)
scripts = root / 'scripts'
scripts.mkdir()
for pattern in ('nao_*.py', 'analysis_visual*.py'):
for path in (ROOT / 'scripts').glob(pattern):
(scripts / path.name).write_text(adapt_script(path.name, path.read_text()), encoding='utf-8')
for name in ('dbgpt_trial_data.py', 'local_api.py', 'project.py'):
shutil.copy2(ROOT / 'scripts' / name, scripts / name)
shutil.copy2(TEMPLATES / 'runtime_platform.py', scripts / 'runtime_platform.py')
policy = root / 'backend/src/main/resources/analysis-runtime-policy.json'
policy.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(ROOT / 'backend/src/main/resources/analysis-runtime-policy.json', policy)
shutil.copytree(ROOT / 'ai/nao-quality', root / 'ai/nao-quality', ignore=shutil.ignore_patterns('.DS_Store'))
deploy = root / 'deploy/offline'
deploy.mkdir(parents=True)
for name in ('control.py', 'gateway.py', 'runtime_platform.py', 'migration_common.py',
'restore_migration.py', 'update_manager.py', 'update-public-key.pem',
'windows-start.ps1', 'windows-prepare.ps1', 'README.md'):
if not windows and name.endswith('.ps1'):
continue
source = TEMPLATES / name
if name.endswith('.ps1'):
(deploy / name).write_text(source.read_text(), encoding='utf-8-sig')
else:
shutil.copy2(source, deploy / name)
shutil.copy2(work / f'requirements-{osname}.lock', root / 'requirements.lock')
if windows:
local_runtime = windows_local_runtime(root)
shutil.copy2(root / 'runtimes/python/python.exe', root / 'runtimes/python/python3.exe')
bootstrap = root / 'deploy/bootstrap'
bootstrap.mkdir()
shutil.copy2(TEMPLATES / 'update_bootstrap.py', bootstrap / 'update-recovery.py')
for name in ('windows-start.ps1', 'windows-prepare.ps1'):
(bootstrap / name).write_text((TEMPLATES / name).read_text(), encoding='utf-8-sig')
(root / '一键部署启动.cmd').write_bytes(
b'@echo off\r\nchcp 65001 >nul\r\npowershell.exe -NoProfile -ExecutionPolicy Bypass -File "%~dp0deploy\\bootstrap\\windows-start.ps1"\r\n'
b'if errorlevel 1 (echo Startup failed. See the message above.)\r\npause\r\n')
for name, action in (('停止服务.cmd', 'stop'), ('查看状态.cmd', 'status'), ('检查环境.cmd', 'verify')):
(root / name).write_bytes(('@echo off\r\nchcp 65001 >nul\r\nset PYTHONUTF8=1\r\nset PYTHONIOENCODING=utf-8\r\n'
'"%~dp0runtimes\\python\\python.exe" "%~dp0deploy\\offline\\control.py" ' + action + '\r\npause\r\n').encode())
else:
libs = root / 'runtimes/linux-libs'
libs.mkdir()
for item in assets:
if item.get('kind') == 'deb':
archive = work / 'downloads' / item['file']
if digest(archive) != item['sha256']:
raise RuntimeError('Debian library checksum mismatch')
extract_deb(archive, libs)
for name, action in (('start.sh', 'start'), ('stop.sh', 'stop'), ('status.sh', 'status'), ('verify.sh', 'verify')):
(root / name).write_text('#!/bin/sh\nset -eu\nROOT=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)\n'
'if [ "$(id -u)" -eq 0 ]; then echo "Run as a normal user, not root." >&2; exit 1; fi\n'
'export PYTHONUTF8=1 PYTHONIOENCODING=utf-8 PYTHONDONTWRITEBYTECODE=1\n'
'export LD_LIBRARY_PATH="$ROOT/runtimes/linux-libs:$ROOT/runtimes/mysql/lib:$ROOT/runtimes/mysql/lib/private"\n'
'exec "$ROOT/runtimes/python/bin/python3" "$ROOT/deploy/offline/control.py" ' + action + ' "$@"\n')
(root / name).chmod(0o755)
system = ('Windows 10/11 或 Windows Server 2019/2022/2025,Intel/AMD x64。仅当前用户运行,无需管理员权限。'
if windows else 'Ubuntu 22.04/24.04、Debian 12/13 的 Intel/AMD x86_64。需要 glibc 2.35 及以上;不支持 Alpine/musl、ARM 或旧版 CentOS。')
usage = ('解压到当前用户的 %LOCALAPPDATA%\\QualityCenter 等较短的本地 NTFS 路径,双击“一键部署启动.cmd”。不要在压缩包中直接打开。\n'
'运行库随包提供;不安装系统服务,不修改系统环境变量、注册表或防火墙,不申请管理员权限。\n'
'启动后自动打开浏览器,命令窗口显示本机和内网地址。关闭此窗口不会停止服务。\n'
'其他电脑连接同一内网后,用浏览器打开显示的 http://内网IP:5280 即可。\n'
'停止服务请双击“停止服务.cmd”;重新启动仍双击原入口。\n' if windows else
'解压后以普通用户运行 ./start.sh;内网共享运行 ./start.sh --lan。\n'
'同事用浏览器打开显示的 http://内网IP:5280;如系统有防火墙,仅对可信内网放行该端口。\n'
'停止运行 ./stop.sh,查看状态运行 ./status.sh,检查完整性运行 ./verify.sh。\n')
(root / '使用说明.txt').write_text('质量中心离线部署包\n\n系统要求\n' + system + '\n建议内存 8GB 以上,至少预留 5GB 磁盘及后续业务数据空间。\n\n'
+ usage + '\n登录信息保存在 runtimedata/登录信息.txt,初始管理员密码在本机随机生成。登录后请修改密码。\n'
'首次数据库为空,不包含原电脑的用户、业务数据、来源文件和模型密钥。通过网页上传授权的数据文件。\n'
'普通功能和安装不需要外网;AI 在线模型调用需要网络,或自行配置内网模型。\n'
'不要在服务运行时移动目录,不要覆盖 .runtime 和 runtimedata。数据迁移使用逻辑备份,不能复制 macOS MySQL 原始数据目录。\n'
'内网访问需要主机保持开机、服务运行、网络互通;本机 MySQL 和内部 API 端口不对内网开放。\n'
'默认网页端口 5280。更改 .runtime/deployment.json 中的 webPort 后重新运行入口。内网连接受现有防火墙规则限制,程序不会自动更改这些规则。\n'
'详细技术说明见 deploy/offline/README.md。\n\n'
'验证范围:前后端隔离构建及本机自动检查已完成;Windows/Linux 原生首次启动和跨电脑内网访问需要在目标机实测。\n', encoding='utf-8-sig')
metadata = {'platform': osname, 'architecture': 'x86_64', 'builtAtUtc': datetime.now(timezone.utc).isoformat(),
'updateVersion': '2026.11.04.1', 'updateSequence': 1, 'installationScope': 'current-user', 'requiresAdministrator': False,
'databaseIncluded': False, 'credentialsIncluded': False, 'requiresDownloadAtInstall': False,
'nativeTargetSmokeTest': 'not-run-on-macos-host',
'runtimes': [{k: v for k, v in a.items() if k not in ('kind',)} for a in assets if a['os'] == osname and a['name'] != 'vcredist'] + ([local_runtime] if windows else []),
'pythonWheels': [{'file': p.name, 'sha256': digest(p)} for p in wheels], 'files': []}
for path in sorted(root.rglob('*')):
if path.is_symlink():
metadata['files'].append({'path': path.relative_to(root).as_posix(), 'symlink': str(path.readlink())})
elif path.is_file():
metadata['files'].append({'path': path.relative_to(root).as_posix(), 'sha256': digest(path)})
(root / 'manifest.json').write_text(json.dumps(metadata, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
return metadata
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--work', type=Path, required=True)
parser.add_argument('--platform', choices=('windows', 'linux', 'both'), default='both')
parser.add_argument('--assemble-only', action='store_true')
args = parser.parse_args()
work = args.work.resolve()
assets = json.loads((work / 'downloads/assets.json').read_text())
platforms = ('windows', 'linux') if args.platform == 'both' else (args.platform,)
summaries = []
for osname in platforms:
name = 'QualityCenter-' + ('Windows' if osname == 'windows' else 'Linux') + '-x64'
directory = work / name
if directory.exists():
raise RuntimeError('Preserving an existing bundle; choose an empty work directory: ' + str(directory))
print('Assembling ' + name, flush=True)
metadata = payload(directory, work, osname, assets)
print('Files verified: ' + str(len(metadata['files'])), flush=True)
if args.assemble_only:
continue
if osname == 'windows':
archive = work / (name + '.zip')
with zipfile.ZipFile(archive, 'w', zipfile.ZIP_DEFLATED, compresslevel=6) as target:
for path in sorted(directory.rglob('*')):
if path.is_file():
target.write(path, path.relative_to(work).as_posix())
else:
archive = work / (name + '.tar.gz')
with tarfile.open(archive, 'w:gz', compresslevel=6) as target:
target.add(directory, arcname=name)
summaries.append({'file': archive.name, 'bytes': archive.stat().st_size, 'sha256': digest(archive)})
print(json.dumps(summaries[-1]), flush=True)
if summaries:
(work / 'SHA256SUMS.txt').write_text(''.join(a['sha256'] + ' ' + a['file'] + '\n' for a in summaries))
(work / 'packages.json').write_text(json.dumps(summaries, indent=2) + '\n')
if __name__ == '__main__':
main()
+486
View File
@@ -0,0 +1,486 @@
#!/usr/bin/env python3
"""Offline, unprivileged service supervisor for the standalone distribution."""
from __future__ import annotations
import argparse
import contextlib
import hashlib
import json
import os
from pathlib import Path
import platform
import re
import secrets
import signal
import socket
import subprocess
import sys
import threading
import time
from urllib.request import ProxyHandler, build_opener
import psutil
ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / 'scripts'))
import runtime_platform as host
from gateway import make_server
PRIVATE = ROOT / '.runtime'
DATA = ROOT / 'runtimedata'
CONFIG = PRIVATE / 'deployment.json'
STATE = PRIVATE / 'service.json'
STOP = PRIVATE / 'stop-request.json'
IS_WINDOWS = os.name == 'nt'
EXE = '.exe' if IS_WINDOWS else ''
PYTHON = ROOT / 'runtimes/python' / ('python.exe' if IS_WINDOWS else 'bin/python3')
JAVA = ROOT / 'runtimes/java/bin' / ('java' + EXE)
MYSQL = ROOT / 'runtimes/mysql/bin' / ('mysql' + EXE)
MYSQLD = ROOT / 'runtimes/mysql/bin' / ('mysqld' + EXE)
ADMIN = ROOT / 'runtimes/mysql/bin' / ('mysqladmin' + EXE)
OPENER = build_opener(ProxyHandler({}))
def atomic(path, value):
temporary = path.with_name(path.name + '.' + secrets.token_hex(5) + '.tmp')
fd = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(fd, 'w', encoding='utf-8') as stream:
json.dump(value, stream, ensure_ascii=False, indent=2)
temporary.replace(path)
def private_directory(path):
if path.is_symlink():
raise RuntimeError('运行目录不能是符号链接。')
path.mkdir(parents=True, exist_ok=True, mode=0o700)
if IS_WINDOWS:
result = subprocess.run(['whoami', '/user', '/fo', 'csv', '/nh'], capture_output=True, check=True)
sid = re.search(rb'S-1-\d+(?:-\d+)+', result.stdout)
if not sid:
raise RuntimeError('无法确定当前 Windows 用户。')
subprocess.run(['icacls', str(path), '/inheritance:r', '/grant:r',
'*' + sid.group().decode() + ':(OI)(CI)F', '*S-1-5-18:(OI)(CI)F', '/Q'],
stdout=subprocess.DEVNULL, check=True)
else:
path.chmod(0o700)
def configuration():
if CONFIG.exists():
value = json.loads(CONFIG.read_text(encoding='utf-8'))
else:
value = {'mysqlPort': 53306, 'backendPort': 5281, 'webPort': 5280,
'webBind': '0.0.0.0' if IS_WINDOWS else '127.0.0.1',
'mysqlPassword': secrets.token_urlsafe(32), 'rootPassword': secrets.token_urlsafe(32),
'adminUsername': 'admin', 'adminPassword': secrets.token_urlsafe(24)}
atomic(CONFIG, value)
value.setdefault('mysqlDatabase', 'quality_platform')
if not re.fullmatch(r'quality_(?:platform|restore_[a-f0-9]{12,32})', value['mysqlDatabase']):
raise RuntimeError('部署数据库名称无效。')
relative_archive = Path(value.get('archiveDirectory', 'runtimedata/archive'))
if relative_archive.is_absolute() or '..' in relative_archive.parts or not (ROOT / relative_archive).resolve().is_relative_to(DATA.resolve()):
raise RuntimeError('来源归档目录无效。')
for key in ('mysqlPort', 'backendPort', 'webPort'):
if type(value.get(key)) is not int or not 1024 <= value[key] <= 65535:
raise RuntimeError('部署端口配置无效。')
if value['backendPort'] not in (5281, 5283, 5284, 5285):
raise RuntimeError('接口端口必须为 5281、5283、5284 或 5285。')
if len({value[k] for k in ('mysqlPort', 'backendPort', 'webPort')}) != 3:
raise RuntimeError('数据库、接口和网页端口不能重复。')
if value.get('webBind') not in ('127.0.0.1', '0.0.0.0'):
raise RuntimeError('网页监听配置无效。')
for key in ('mysqlPassword', 'rootPassword', 'adminPassword'):
if not re.fullmatch(r'[A-Za-z0-9_-]{20,100}', value.get(key, '')):
raise RuntimeError('本机随机凭据配置无效,原配置已保留。')
return value
def environment(value):
env = os.environ.copy()
package = ROOT / 'runtimes/python' / ('Lib/site-packages' if IS_WINDOWS else 'lib/python3.11/site-packages') / 'nao_core'
env.update({
'JAVA_HOME': str(JAVA.parent.parent),
'PATH': os.pathsep.join([str(PYTHON.parent), str(JAVA.parent), str(MYSQL.parent), env.get('PATH', '')]),
'PYTHONUTF8': '1', 'PYTHONIOENCODING': 'utf-8', 'PYTHONDONTWRITEBYTECODE': '1',
'QUALITY_PROJECT_ROOT': str(ROOT), 'QUALITY_ASSISTANT_PYTHON': str(PYTHON),
'QUALITY_ASSISTANT_PACKAGE': str(package),
'MYSQL_URL': f'jdbc:mysql://127.0.0.1:{value["mysqlPort"]}/{value.get("mysqlDatabase", "quality_platform")}?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC&forceConnectionTimeZoneToSession=true&useSSL=false&allowPublicKeyRetrieval=true',
'MYSQL_USER': 'quality_app', 'MYSQL_PASSWORD': value['mysqlPassword'],
'QUALITY_BOOTSTRAP_ADMIN_USERNAME': '' if value.get('restoredFromMigration') else value['adminUsername'],
'QUALITY_BOOTSTRAP_ADMIN_PASSWORD': '' if value.get('restoredFromMigration') else value['adminPassword'],
'QUALITY_ARCHIVE_ROOT': str(ROOT / value.get('archiveDirectory', 'runtimedata/archive')), 'QUALITY_ALLOWED_ROOTS': str(DATA / 'sources'),
'QUALITY_ANALYSIS_RUNTIME_QUERY_PORT': str(value['backendPort']),
'PORT': str(value['backendPort']), 'SERVER_ADDRESS': '127.0.0.1',
'QUALITY_DATA_WORKFLOW_TEST_MODE': 'false',
'MYSQL_LEGACY_TIMESTAMP_OFFSET_SECONDS': 'UNCONFIRMED',
})
env['QUALITY_DATA_WORKFLOW_TEST_MODE'] = str(value.get('dataWorkflowTestMode', False)).lower()
env['QUALITY_ANALYSIS_AUTONOMOUS_ENABLED'] = str(value.get('analysisAutonomousEnabled', False)).lower()
env['QUALITY_ANALYSIS_PROTOCOL_V1_ENABLED'] = str(value.get('analysisProtocolV1Enabled', False)).lower()
env['QUALITY_UPDATE_ENABLED'] = 'true' if IS_WINDOWS else 'false'
# Do not accidentally inherit a development or source-machine AI command.
env.pop('QUALITY_ANALYSIS_NAO_COMMAND', None)
env.pop('QUALITY_ANALYSIS_NAO_SCRIPT', None)
env.pop('PYTHONHOME', None)
env.pop('PYTHONPATH', None)
if not IS_WINDOWS:
env['LD_LIBRARY_PATH'] = os.pathsep.join([str(ROOT / 'runtimes/linux-libs'),
str(ROOT / 'runtimes/mysql/lib'),
str(ROOT / 'runtimes/mysql/lib/private')])
return env
def record(process, command):
observed = psutil.Process(process.pid)
return {'pid': process.pid, 'created': observed.create_time(),
'executable': str(Path(observed.exe()).resolve()), 'args': [str(a) for a in command[1:]]}
def matches(info):
try:
process = psutil.Process(int(info['pid']))
return (process.is_running() and process.status() != psutil.STATUS_ZOMBIE
and abs(process.create_time() - info['created']) < .01
and os.path.normcase(str(Path(process.exe()).resolve())) == os.path.normcase(info['executable'])
and process.cmdline()[1:] == info['args'])
except (psutil.Error, KeyError, ValueError, TypeError):
return False
def read_state():
try:
return json.loads(STATE.read_text(encoding='utf-8'))
except FileNotFoundError:
return {}
def free_port(port, bind='127.0.0.1'):
# A recently closed TCP connection may remain in TIME_WAIT. Reject a live
# listener, but allow a clean restart without waiting for that state to expire.
try:
connection = socket.create_connection(('127.0.0.1', port), timeout=.3)
except OSError:
pass
else:
connection.close()
raise RuntimeError(f'端口 {port} 已被其他服务使用。')
with socket.socket() as test:
test.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
test.bind((bind, port))
def mysql_command(value, binary=MYSQL):
return [str(binary), '--no-defaults', '--protocol=TCP', '--host=127.0.0.1',
'--port=' + str(value['mysqlPort']), '--user=root', '--connect-timeout=3']
def sql(value, query):
env = environment(value)
env['MYSQL_PWD'] = value['rootPassword']
return subprocess.run(mysql_command(value) + ['--batch', '--skip-column-names'], input=query,
text=True, encoding='utf-8', stdout=subprocess.PIPE, stderr=subprocess.PIPE,
env=env, timeout=10)
def verify(check_hashes=True):
manifest = json.loads((ROOT / 'manifest.json').read_text(encoding='utf-8'))
expected_os = 'windows' if IS_WINDOWS else platform.system().lower()
if manifest['platform'] != expected_os or platform.machine().lower() not in ('amd64', 'x86_64'):
raise RuntimeError('此包仅用于 ' + manifest['platform'] + ' 的 Intel/AMD 64 位电脑。')
if check_hashes:
for entry in manifest['files']:
path = ROOT / entry['path']
if entry.get('symlink') is not None:
valid = path.is_symlink() and os.readlink(path) == entry['symlink']
else:
valid = path.is_file() and hashlib.file_digest(path.open('rb'), 'sha256').hexdigest() == entry['sha256']
if not valid:
raise RuntimeError('文件缺失或校验失败:' + entry['path'])
value = configuration()
env = environment(value)
for command in ([JAVA, '-version'], [MYSQLD, '--no-defaults', '--version']):
result = subprocess.run([str(a) for a in command], capture_output=True, env=env, timeout=30)
if result.returncode:
raise RuntimeError('运行环境检查失败:' + Path(command[0]).name + '。请查看安装说明中的系统要求。')
probe = subprocess.run([str(PYTHON), '-c', 'import duckdb,yaml,fastapi,psutil,nao_core;import sys;assert sys.version_info[:3]==(3,11,15)'],
capture_output=True, env=env, timeout=45)
if probe.returncode:
raise RuntimeError('AI Python 运行环境检查失败。')
application_probe = subprocess.run([str(PYTHON), '-c',
'import sys;sys.path.insert(0,sys.argv[1]);import nao_quality_runtime as r;'
'import nao_native_run,nao_quality_runtime_mcp,nao_visual_recovery;'
'assert len(r._config_fingerprint("https://offline.invalid/v1","offline-check","offline-check"))==64',
str(ROOT / 'scripts')], capture_output=True, env=env, timeout=45)
if application_probe.returncode:
raise RuntimeError('AI 应用脚本或策略资源检查失败。')
package = Path(env['QUALITY_ASSISTANT_PACKAGE'])
native = subprocess.run([str(package / 'bin' / ('nao-chat-server' + EXE)), '--help'],
capture_output=True, env=env, timeout=30)
if native.returncode:
raise RuntimeError('AI 原生服务无法运行,请检查系统版本和 CPU 支持。')
return value
def addresses(value):
ips = set()
if value['webBind'] == '0.0.0.0':
for entries in psutil.net_if_addrs().values():
for entry in entries:
if entry.family == socket.AF_INET and not entry.address.startswith(('127.', '169.254.')):
ips.add(entry.address)
return ['http://127.0.0.1:' + str(value['webPort'])] + [f'http://{ip}:{value["webPort"]}' for ip in sorted(ips)]
def write_login(value):
path = DATA / '登录信息.txt'
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, 'w', encoding='utf-8') as stream:
if value.get('restoredFromMigration'):
stream.write('质量中心\n' + '\n'.join(addresses(value)) + '\n业务数据已迁移,请使用原有账号和密码登录。\n')
else:
stream.write('质量中心\n' + '\n'.join(addresses(value)) + '\n用户名:' + value['adminUsername']
+ '\n初始密码:' + value['adminPassword'] + '\n登录后请在个人账户修改密码。此文件仅记录初始密码,请勿分享。\n')
def serve(database_only=False):
lock = os.open(PRIVATE / 'supervisor.lock', os.O_RDWR | os.O_CREAT, 0o600)
try:
host.fcntl.flock(lock, host.fcntl.LOCK_EX | host.fcntl.LOCK_NB)
except BlockingIOError:
raise RuntimeError('此目录的服务已经启动。') from None
children = []
value = configuration()
state = {'supervisor': record(psutil.Process(), [sys.executable, *sys.argv]),
'token': secrets.token_hex(24), 'ready': False, 'databaseOnly': database_only, 'children': children}
stopped = threading.Event()
for sig in (signal.SIGINT, signal.SIGTERM):
signal.signal(sig, lambda *_: stopped.set())
def stopping():
if STOP.exists():
try:
if json.loads(STOP.read_text(encoding='utf-8')).get('token') == state['token']:
stopped.set()
except (OSError, ValueError):
pass
return stopped.is_set()
def spawn(name, command):
with (DATA / 'logs' / (name + '.log')).open('ab') as log:
child = subprocess.Popen([str(a) for a in command], cwd=ROOT, env=environment(value),
stdin=subprocess.DEVNULL, stdout=log, stderr=log)
info = record(child, command)
info['name'] = name
children.append(info)
atomic(STATE, state)
return child
def wait_ready(child, check, seconds):
end = time.monotonic() + seconds
while time.monotonic() < end:
if stopping():
raise RuntimeError('启动已停止。')
if child.poll() is not None:
raise RuntimeError('服务启动失败,请查看 runtimedata/logs。')
try:
if check():
return
except (OSError, ValueError, subprocess.TimeoutExpired):
pass
time.sleep(.5)
raise RuntimeError('服务启动超时,请查看 runtimedata/logs。')
try:
for key in (('mysqlPort',) if database_only else ('mysqlPort', 'backendPort', 'webPort')):
free_port(value[key], value['webBind'] if key == 'webPort' else '127.0.0.1')
atomic(STATE, state)
db = DATA / 'mysql'
if not (db / 'auto.cnf').exists():
if db.exists() and any(db.iterdir()):
raise RuntimeError('数据库目录含有未完成的初始化文件,已保留原目录;请查看初始化日志。')
db.mkdir(exist_ok=True)
with (DATA / 'logs/mysql-initialize.log').open('ab') as log:
result = subprocess.run([str(MYSQLD), '--no-defaults', '--initialize-insecure',
'--basedir=' + str(MYSQLD.parent.parent), '--datadir=' + str(db),
'--lower-case-table-names=1'], env=environment(value),
stdout=log, stderr=log, timeout=180)
if result.returncode:
raise RuntimeError('数据库初始化失败,请查看 runtimedata/logs/mysql-initialize.log。')
bootstrap = PRIVATE / 'bootstrap.sql'
fd = os.open(bootstrap, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, 'w', encoding='utf-8') as stream:
stream.write(f"ALTER USER 'root'@'localhost' IDENTIFIED BY '{value['rootPassword']}';\n"
f"CREATE USER IF NOT EXISTS 'root'@'127.0.0.1' IDENTIFIED BY '{value['rootPassword']}';\n"
"GRANT ALL PRIVILEGES ON *.* TO 'root'@'127.0.0.1' WITH GRANT OPTION;\n"
f"CREATE DATABASE IF NOT EXISTS `{value['mysqlDatabase']}` CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;\n"
f"CREATE USER IF NOT EXISTS 'quality_app'@'127.0.0.1' IDENTIFIED BY '{value['mysqlPassword']}';\n"
f"GRANT ALL PRIVILEGES ON `{value['mysqlDatabase']}`.* TO 'quality_app'@'127.0.0.1';\n")
mysql_args = [MYSQLD, '--no-defaults', '--basedir=' + str(MYSQLD.parent.parent), '--datadir=' + str(db),
'--port=' + str(value['mysqlPort']), '--bind-address=127.0.0.1', '--mysqlx=OFF',
'--lower-case-table-names=1', '--skip-name-resolve', '--skip-log-bin',
'--max-connections=60', '--innodb-buffer-pool-size=' + ('1024M' if database_only else '256M'), '--init-file=' + bootstrap.as_posix()]
if database_only:
# The restored database is unpublished until validation and a clean
# MySQL shutdown finish. Normal startup restores durable defaults.
mysql_args += ['--innodb-redo-log-capacity=1G', '--innodb-flush-log-at-trx-commit=2']
if not IS_WINDOWS:
mysql_args.append('--socket=' + str(PRIVATE / 'mysql.sock'))
database = spawn('mysql', mysql_args)
wait_ready(database, lambda: sql(value, 'SELECT 1;').returncode == 0, 120)
# The server does not reread this file after startup.
bootstrap.unlink(missing_ok=True)
if database_only:
state['ready'] = True
atomic(STATE, state)
while not stopping():
if database.poll() is not None:
raise RuntimeError('恢复数据库服务已退出,请查看日志。')
time.sleep(.25)
return
backend = spawn('backend', [JAVA, '-Djava.awt.headless=true', '-Xms128m', '-Xmx2048m', '-jar', ROOT / 'app/backend.jar'])
def backend_ready():
with OPENER.open(f'http://127.0.0.1:{value["backendPort"]}/actuator/health', timeout=3) as response:
return json.load(response).get('status') == 'UP'
wait_ready(backend, backend_ready, 900)
server = make_server(ROOT / 'app/web', value['webBind'], value['webPort'], value['backendPort'])
server.timeout = .5
state['ready'] = True
write_login(value)
atomic(STATE, state)
print('质量中心已启动:' + ' '.join(addresses(value)), flush=True)
try:
while not stopping():
if database.poll() is not None or backend.poll() is not None:
raise RuntimeError('后台服务已退出,请查看 runtimedata/logs。')
server.handle_request()
finally:
server.server_close()
finally:
state['ready'] = False
atomic(STATE, state)
# Stop backend first, then its isolated AI processes, then the database.
for info in reversed(children):
if info['name'] != 'backend' or not matches(info):
continue
process = psutil.Process(info['pid'])
descendants = process.children(recursive=True)
process.terminate()
try:
process.wait(35)
except psutil.TimeoutExpired:
if matches(info):
process.kill()
for child in descendants:
with contextlib.suppress(psutil.NoSuchProcess):
child.terminate()
with contextlib.suppress(OSError, subprocess.SubprocessError):
subprocess.run([str(PYTHON), str(ROOT / 'scripts/nao_quality_runtime.py'), '--stop-all'],
env=environment(value), cwd=ROOT, capture_output=True, timeout=25)
if any(info['name'] == 'mysql' and matches(info) for info in children):
env = environment(value)
env['MYSQL_PWD'] = value['rootPassword']
with contextlib.suppress(OSError, subprocess.SubprocessError):
subprocess.run(mysql_command(value, ADMIN) + ['shutdown'], env=env,
capture_output=True, timeout=60)
# mysqladmin can return before InnoDB has finished closing files.
for info in children:
if info['name'] == 'mysql' and matches(info):
with contextlib.suppress(psutil.NoSuchProcess, psutil.TimeoutExpired):
psutil.Process(info['pid']).wait(60)
survivors = [info for info in children if matches(info)]
if survivors:
state['children'] = survivors
atomic(STATE, state)
else:
STATE.unlink(missing_ok=True)
STOP.unlink(missing_ok=True)
(PRIVATE / 'bootstrap.sql').unlink(missing_ok=True)
os.close(lock)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('action', choices=('start', 'stop', 'status', 'verify', 'serve'))
parser.add_argument('--lan', action='store_true')
parser.add_argument('--database-only', action='store_true', help=argparse.SUPPRESS)
args = parser.parse_args()
if args.action in ('start', 'serve', 'verify'):
private_directory(PRIVATE)
private_directory(DATA)
(DATA / 'logs').mkdir(exist_ok=True)
(DATA / 'sources').mkdir(exist_ok=True)
(DATA / 'archive').mkdir(exist_ok=True)
if args.action == 'serve':
return serve(args.database_only)
updater = ROOT / 'deploy/offline/update_manager.py'
if args.action == 'start' and IS_WINDOWS and updater.is_file():
recovered = subprocess.run([str(PYTHON), str(updater), 'recover'], cwd=ROOT,
env=environment(configuration()), capture_output=True, text=True,
encoding='utf-8')
if recovered.returncode:
try:
message = json.loads(recovered.stdout).get('error')
except (ValueError, AttributeError):
message = None
raise RuntimeError(message or '程序更新正在进行或尚未安全恢复,请查看更新记录。')
state = read_state()
running = matches(state.get('supervisor', {}))
if args.action == 'status':
print('已启动' if running and state.get('ready') else '正在启动或停止' if running else '已停止')
return
if args.action == 'stop':
if not running:
if any(matches(info) for info in state.get('children', [])):
raise RuntimeError('发现上次退出后遗留的进程,请根据日志检查;未结束任何不明进程。')
print('服务已停止。')
return
atomic(STOP, {'token': state['token']})
for _ in range(240):
if not matches(state['supervisor']):
print('服务已停止。')
return
time.sleep(.5)
raise RuntimeError('服务仍在安全关闭,请稍后检查状态。')
if running:
if args.lan and configuration()['webBind'] != '0.0.0.0':
raise RuntimeError('请先停止服务,再使用 --lan 启动。')
print('服务已启动。' if state.get('ready') else '服务正在启动,请稍候。')
for url in addresses(configuration()):
print(url)
return
if any(matches(info) for info in state.get('children', [])):
raise RuntimeError('发现上次退出后的服务进程,原数据已保留;请检查日志。')
value = verify()
if args.action == 'verify':
print('文件完整性、Java、MySQL、AI Python 与 AI 原生程序检查通过。')
return
if args.lan and value['webBind'] != '0.0.0.0':
value['webBind'] = '0.0.0.0'
atomic(CONFIG, value)
with (DATA / 'logs/supervisor.log').open('ab') as log:
options = {'creationflags': subprocess.DETACHED_PROCESS | subprocess.CREATE_NEW_PROCESS_GROUP} if IS_WINDOWS else {'start_new_session': True}
process = subprocess.Popen([str(PYTHON), str(Path(__file__).resolve()), 'serve'], cwd=ROOT,
env=environment(value), stdin=subprocess.DEVNULL, stdout=log, stderr=log, **options)
for _ in range(2000):
state = read_state()
if state.get('ready') and state.get('supervisor', {}).get('pid') == process.pid and matches(state['supervisor']):
for url in addresses(value):
print(url)
print('管理员初始登录信息:' + str(DATA / '登录信息.txt'))
if IS_WINDOWS:
os.startfile(addresses(value)[0])
return
if process.poll() is not None:
raise RuntimeError('启动未完成,请查看 runtimedata/logs/supervisor.log。')
time.sleep(.5)
raise RuntimeError('服务仍在启动,请检查状态与日志。')
if __name__ == '__main__':
try:
main()
except (RuntimeError, OSError, subprocess.SubprocessError) as error:
print(str(error), file=sys.stderr)
raise SystemExit(1)
+102
View File
@@ -0,0 +1,102 @@
"""Serve built assets and stream the authenticated API on one LAN port."""
from __future__ import annotations
import functools
import http.client
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from urllib.parse import urlsplit
HOP_HEADERS = {'connection', 'keep-alive', 'proxy-authenticate', 'proxy-authorization',
'te', 'trailer', 'transfer-encoding', 'upgrade'}
class Gateway(SimpleHTTPRequestHandler):
protocol_version = 'HTTP/1.1'
backend_port = 5281
extensions_map = {**SimpleHTTPRequestHandler.extensions_map, '.js': 'text/javascript',
'.mjs': 'text/javascript', '.css': 'text/css', '.svg': 'image/svg+xml',
'.wasm': 'application/wasm'}
def log_message(self, *_):
# Do not record query strings, authentication data or business values.
pass
def _proxy(self):
upstream = http.client.HTTPConnection('127.0.0.1', self.backend_port, timeout=1800)
try:
if self.headers.get('Transfer-Encoding'):
self.send_error(400, 'Content-Length is required')
return
length = int(self.headers.get('Content-Length', '0'))
if length < 0 or length > 273 * 1024 * 1024:
self.send_error(413)
return
upstream.putrequest(self.command, self.path, skip_host=True, skip_accept_encoding=True)
connection_headers = {s.strip().lower() for s in self.headers.get('Connection', '').split(',')}
for name, value in self.headers.items():
if name.lower() not in HOP_HEADERS | connection_headers:
upstream.putheader(name, value)
upstream.putheader('Connection', 'close')
upstream.endheaders()
remaining = length
while remaining:
chunk = self.rfile.read(min(remaining, 1024 * 1024))
if not chunk:
raise ConnectionError('Incomplete upload')
upstream.send(chunk)
remaining -= len(chunk)
response = upstream.getresponse()
self.send_response(response.status)
response_connection = {s.strip().lower() for s in response.getheader('Connection', '').split(',')}
for name, value in response.getheaders():
if name.lower() not in HOP_HEADERS | response_connection:
self.send_header(name, value)
self.send_header('Connection', 'close')
self.send_header('X-Content-Type-Options', 'nosniff')
self.end_headers()
self.close_connection = True
if self.command != 'HEAD':
while chunk := response.read1(64 * 1024):
self.wfile.write(chunk)
self.wfile.flush() # SSE events must arrive without buffering.
except (BrokenPipeError, ConnectionResetError):
self.close_connection = True
except (OSError, http.client.HTTPException, ValueError):
self.close_connection = True
# Never expose upstream exception text or an internal file path.
if not self.wfile.closed:
try:
self.send_error(502, 'Backend unavailable')
except OSError:
pass
finally:
upstream.close()
def _dispatch(self):
path = urlsplit(self.path).path
if path == '/api' or path.startswith('/api/') or path == '/actuator/health':
return self._proxy()
if self.command not in ('GET', 'HEAD'):
self.send_error(405)
return
candidate = Path(self.translate_path(self.path)).resolve()
root = Path(self.directory).resolve()
if not candidate.is_relative_to(root):
self.send_error(403)
return
if candidate.is_dir() and not (candidate / 'index.html').is_file():
self.send_error(404)
return
if not candidate.exists() and not Path(path).suffix:
self.path = '/index.html'
return super().do_HEAD() if self.command == 'HEAD' else super().do_GET()
do_GET = do_HEAD = do_POST = do_PUT = do_PATCH = do_DELETE = do_OPTIONS = _dispatch
def make_server(root, address, port, backend_port=5281):
handler = type('ConfiguredGateway', (Gateway,), {'backend_port': backend_port})
server = ThreadingHTTPServer((address, port), functools.partial(handler, directory=str(root)))
server.daemon_threads = True
return server
+154
View File
@@ -0,0 +1,154 @@
"""Portable migration validation; no source-machine paths or credentials are logged."""
from __future__ import annotations
import gzip
import hashlib
import json
from pathlib import Path, PurePosixPath, PureWindowsPath
import re
import tarfile
import uuid
def sha256(path):
with Path(path).open('rb') as stream:
return hashlib.file_digest(stream, 'sha256').hexdigest()
def sql_text(value):
return 'CONVERT(0x' + value.encode('utf-8').hex() + ' USING utf8mb4)'
def relative_name(name):
if not isinstance(name, str) or not name or '\\' in name or ':' in name:
raise ValueError('迁移文件路径无效。')
path = PurePosixPath(name)
if not path.parts or path.is_absolute() or path.as_posix() != name or any(part in ('.', '..', '') for part in path.parts):
raise ValueError('迁移文件路径无效。')
for part in path.parts:
if part.endswith((' ', '.')) or re.fullmatch(r'(?i)(CON|PRN|AUX|NUL|COM[1-9]|LPT[1-9])(?:\..*)?', part):
raise ValueError('迁移文件名不兼容 Windows。')
return path
def path_under_source(stored, source_root):
# A Windows restore must interpret the old Mac path as POSIX, not Path().
path, root = PurePosixPath(stored), PurePosixPath(source_root)
if not path.is_absolute() or '..' in path.parts or not path.is_relative_to(root):
raise ValueError('来源文件引用不在已验证的归档目录内。')
return relative_name(path.relative_to(root).as_posix()).as_posix()
def extract_checked(archive, destination, expected):
destination = Path(destination)
destination.mkdir(parents=True, exist_ok=False, mode=0o700)
observed = set()
for name in expected:
relative_name(name)
folded = [name.casefold() for name in expected]
if len(folded) != len(set(folded)):
raise ValueError('迁移文件在 Windows 上存在重名。')
with tarfile.open(archive, 'r:gz') as source:
for member in source:
rel = relative_name(member.name)
if not member.isfile() or member.name in observed or member.name not in expected:
raise ValueError('迁移归档包含未登记或无效的文件。')
entry = expected[member.name]
target = destination.joinpath(*rel.parts)
target.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
if not target.resolve().is_relative_to(destination.resolve()):
raise ValueError('迁移文件路径越界。')
content = source.extractfile(member)
if content is None:
raise ValueError('迁移文件无法读取。')
digest = hashlib.sha256()
with content, target.open('xb') as output:
while chunk := content.read(1024 * 1024):
digest.update(chunk)
output.write(chunk)
target.chmod(0o600)
if target.stat().st_size != entry['bytes'] or digest.hexdigest() != entry['sha256']:
raise ValueError('迁移文件完整性检查失败。')
observed.add(member.name)
if observed != set(expected):
raise ValueError('迁移归档缺少文件。')
def table_counts(query):
result = {}
for name in query('SHOW TABLES;').splitlines():
if not re.fullmatch(r'[A-Za-z0-9_]{1,64}', name):
raise ValueError('数据库表名无效。')
result[name] = int(query('SELECT COUNT(*) FROM `' + name + '`;').strip())
return result
def query_evidence(query):
current = query("SELECT release_id FROM current_release WHERE dataset_id='quality';").strip()
if not current:
return {'releaseId': None, 'months': []}
if not re.fullmatch(r'[A-Za-z0-9-]{1,80}', current):
raise ValueError('正式数据版本无效。')
result = {'releaseId': current, 'months': []}
components = ['black_responsibility_scrap', 'black_material_scrap', 'black_external_scrap',
'bright_headquarters_scrap', 'bright_plating_scrap', 'bright_material_scrap',
'bright_processing_scrap', 'bright_diffusion_scrap']
for stage in ('black', 'bright'):
pairs = "'stage','" + stage + "','month',f.month_key,'rows',COUNT(*),'qualified',SUM(f.qualified_weight),'scrap',SUM(f.scrap_weight),'total',SUM(f.total_weight)"
for key in components:
pairs += ", '" + key + "',SUM(CAST(NULLIF(JSON_UNQUOTE(JSON_EXTRACT(f.components,'$." + key + "')),'null') AS DECIMAL(28,10)))"
result['months'].extend(json.loads(line) for line in query("SELECT JSON_OBJECT(" + pairs + ") FROM fact_" + stage + " f WHERE EXISTS(SELECT 1 FROM release_member rm WHERE rm.release_id='" + current + "' AND rm.transform_build_id=f.build_id) GROUP BY f.month_key ORDER BY f.month_key;").splitlines())
result['members'] = [json.loads(line) for line in query("SELECT JSON_OBJECT('sourceDocumentId',m.source_document_id,'sourceRevisionId',m.source_revision_id,'buildId',m.transform_build_id,'sourceSha256',r.sha256) FROM release_member m JOIN source_revision r ON r.id=m.source_revision_id WHERE m.release_id='" + current + "' ORDER BY m.source_document_id;").splitlines()]
return result
_VALUES_TOKENS = re.compile(rb"'(?:[^'\\]+|\\.)*'|(\),\()")
def inserted_rows(line):
if not line.startswith(b'INSERT INTO `') or b' VALUES (' not in line:
raise ValueError('不支持的数据库导出行。')
values = line.split(b' VALUES ', 1)[1]
if not values.rstrip().endswith(b';'):
raise ValueError('数据库导出行不完整。')
return 1 + sum(match[1] is not None for match in _VALUES_TOKENS.finditer(values))
def dump_counts(path):
counts, uncompressed = {}, 0
with gzip.open(path, 'rb') as source:
for line in source:
uncompressed += len(line)
create = re.match(rb'^CREATE TABLE `([A-Za-z0-9_]+)`', line)
if create:
counts[create[1].decode()] = 0
insert = re.match(rb'^INSERT INTO `([A-Za-z0-9_]+)`', line)
if insert:
name = insert[1].decode()
if name not in counts:
raise ValueError('导出数据缺少表结构。')
counts[name] += inserted_rows(line)
return counts, uncompressed
def filter_models(value):
profiles = value.get('profiles')
if not isinstance(profiles, list):
profiles = [{**value, 'id': 'migrated-profile', 'name': '迁移模型'}]
def excluded(profile):
return any('mimo' in str(profile.get(key, '')).lower() for key in ('model', 'name', 'baseUrl'))
kept = [profile.copy() for profile in profiles if not excluded(profile)]
removed = [profile for profile in profiles if excluded(profile)]
if not kept:
return None, removed
selected = next((profile for profile in kept if profile.get('id') == value.get('activeProfileId')), kept[0])
required = ('baseUrl', 'apiKey', 'model')
if any(not isinstance(selected.get(key), str) or not selected[key] for key in required):
raise ValueError('需要迁移的模型配置不完整。')
filtered = {key: value[key] for key in ('schemaVersion',) if key in value}
filtered.update(selected)
filtered.pop('id', None)
filtered.pop('name', None)
filtered.update({'schemaVersion': 3, 'profiles': kept, 'activeProfileId': selected['id']})
filtered['revision'] = value.get('revision') or str(uuid.uuid4())
return filtered, removed
+200
View File
@@ -0,0 +1,200 @@
"""Capture verified source files and private model settings for a Windows migration."""
from __future__ import annotations
import argparse
from datetime import datetime, timezone
import json
import os
from pathlib import Path
import re
import shutil
import sqlite3
import sys
import tarfile
import uuid
from migration_common import filter_models, relative_name, sha256
ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT / 'scripts'))
import backup
def private_json(path, data):
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, 'w', encoding='utf-8') as output:
json.dump(data, output, ensure_ascii=False, indent=2)
output.write('\n')
def native_copy(source, target):
source_db = sqlite3.connect(source.resolve().as_uri() + '?mode=ro', uri=True)
destination = sqlite3.connect(target)
try:
source_db.backup(destination)
destination.execute('PRAGMA foreign_keys=OFF')
destination.execute('PRAGMA secure_delete=ON')
tables = {row[0] for row in destination.execute("SELECT name FROM sqlite_master WHERE type='table'")}
# Access sessions, external integrations and cached provider keys are
# recreated on Windows. Historical chats and their ownership stay intact.
for name in ('session', 'verification', 'api_key', 'jwks', 'oauth_access_token',
'oauth_refresh_token', 'oauth_consent', 'oauth_client', 'mcp_user_token',
'mcp_oauth_client', 'sandbox_secret', 'log', 'mcp_call_log'):
if name in tables:
destination.execute('DELETE FROM "' + name + '"')
if 'project_llm_config' in tables:
destination.execute('DELETE FROM project_llm_config')
for name in ('user', 'account', 'organization'):
if name not in tables:
continue
columns = list(destination.execute('PRAGMA table_info("' + name + '")'))
for column in columns:
key = column[1]
if key in ('github_access_token', 'gitlab_access_token', 'access_token', 'refresh_token',
'id_token', 'google_client_secret'):
replacement = "''" if column[3] else 'NULL'
destination.execute('UPDATE "' + name + '" SET "' + key + '"=' + replacement)
destination.commit()
destination.execute('VACUUM')
assert destination.execute('PRAGMA integrity_check').fetchone()[0] == 'ok'
finally:
destination.close()
source_db.close()
target.chmod(0o600)
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--work', type=Path, required=True)
args = parser.parse_args()
work = args.work.resolve()
payload = work / 'migration'
manifest = json.loads((payload / 'source-snapshot.json').read_text())
database = manifest['sourceDatabase']
query = lambda sql: backup.client(sql, database)
archive_root = ROOT / 'runtimedata/archive'
files, sources, digest_index = {}, {}, {}
for path in sorted(archive_root.rglob('*')):
if path.is_symlink():
raise RuntimeError('来源归档包含符号链接。')
if not path.is_file():
continue
rel = relative_name(path.relative_to(archive_root).as_posix()).as_posix()
digest = sha256(path)
files[rel] = {'sha256': digest, 'bytes': path.stat().st_size}
sources[rel] = path
digest_index.setdefault(digest, rel)
bindings = []
missing_optional = 0
for table, column, state in (
('source_revision', 'archive_path', "'REQUIRED'"),
('source_candidate', 'source_path', 'status'),
('source_candidate', 'archive_path', 'status'),
('source_upload_draft', 'source_path', "IF(expires_at<CURRENT_TIMESTAMP(6),'EXPIRED','ACTIVE')"),
):
rows = query("SELECT JSON_OBJECT('id',id,'path'," + column + ",'sha256',sha256,'size',file_size,'state'," + state + ') FROM ' + table + ' WHERE ' + column + ' IS NOT NULL;')
for line in rows.splitlines():
row = json.loads(line)
old = Path(row['path'])
digest = row['sha256']
relative = None
if old.is_relative_to(archive_root) and old.is_file():
relative = old.relative_to(archive_root).as_posix()
elif digest in digest_index:
relative = digest_index[digest]
elif old.is_file() and not old.is_symlink():
if sha256(old) != digest or old.stat().st_size != int(row['size']):
raise RuntimeError('需要迁移的来源文件与登记内容不一致。')
relative = 'migration-inputs/' + digest + '.xlsx'
files[relative] = {'sha256': digest, 'bytes': old.stat().st_size}
sources[relative] = old
digest_index[digest] = relative
optional = False
if relative is None:
if table != 'source_upload_draft' or row['state'] != 'EXPIRED':
raise RuntimeError('需要迁移的来源文件缺失。')
relative = 'expired-upload-drafts/' + row['id'] + '.xlsx'
optional = True
missing_optional += 1
else:
info = files[relative]
if info['sha256'] != digest or info['bytes'] != int(row['size']):
raise RuntimeError('来源引用与归档校验不一致。')
bindings.append({'table': table, 'column': column, 'id': row['id'], 'oldPath': row['path'],
'relative': relative, 'sha256': digest, 'bytes': int(row['size']), 'expiredMissing': optional})
archive = payload / 'archive.tar.gz'
with tarfile.open(archive, 'w:gz', compresslevel=1) as output:
for rel, path in sorted(sources.items()):
output.add(path, arcname=rel, recursive=False)
if sha256(path) != files[rel]['sha256']:
raise RuntimeError('打包期间来源文件发生变化。')
archive.chmod(0o600)
print('Source files captured:', len(files), 'references:', len(bindings), flush=True)
config = json.loads((ROOT / '.runtime/nao-model-config.json').read_text())
filtered, excluded = filter_models(config)
if filtered:
private_json(payload / 'model-config.json', filtered)
kept_keys = {p.get('apiKey') for p in (filtered or {}).get('profiles', [])}
forbidden_keys = {p.get('apiKey') for p in excluded if p.get('apiKey') and p.get('apiKey') not in kept_keys}
# Used only by the local privacy verifier; never shipped or printed.
private_json(work / 'excluded-model-audit-private.json', {'keys': sorted(forbidden_keys)})
session_rows = [json.loads(line) for line in query("SELECT JSON_OBJECT('id',id,'userId',user_id) FROM analysis_session;").splitlines()]
users = {row['id'] for row in manifest['sourceUsers']}
native_source = ROOT / '.runtime/nao-users'
native_stage = work / 'native-snapshot'
native_stage.mkdir(mode=0o700, exist_ok=False)
runtime_roots = []
for actor in users:
if not re.fullmatch(r'[a-fA-F0-9-]{36}', actor):
raise RuntimeError('用户标识无效。')
if (native_source / actor / 'private/nao.sqlite').is_file():
runtime_roots.append(native_source / actor)
for row in session_rows:
source = native_source / row['userId'] / 'sessions' / row['id']
if (source / 'private/nao.sqlite').is_file():
runtime_roots.append(source)
selected = set()
for runtime in runtime_roots:
for name in ('credentials.json', 'nao.sqlite', 'native-chat.json', 'current-session.json', 'sessions.json'):
source = runtime / 'private' / name
if source.is_file() and not source.is_symlink():
selected.add(source)
evidence = runtime / 'evidence'
if evidence.is_dir():
selected.update(p for p in evidence.rglob('*') if p.is_file() and not p.is_symlink())
native_files = {}
for source in sorted(selected):
rel = source.relative_to(native_source).as_posix()
relative_name(rel)
target = native_stage / rel
target.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
if source.name == 'nao.sqlite':
native_copy(source, target)
else:
shutil.copy2(source, target)
target.chmod(0o600)
content = target.read_bytes()
if any(key.encode() in content for key in forbidden_keys):
raise RuntimeError('原生会话中检测到被排除模型的密钥,已停止打包。')
native_files[rel] = {'sha256': sha256(target), 'bytes': target.stat().st_size}
with tarfile.open(payload / 'native-history.tar.gz', 'w:gz', compresslevel=1) as output:
for rel in sorted(native_files):
output.add(native_stage / rel, arcname=rel, recursive=False)
(payload / 'native-history.tar.gz').chmod(0o600)
manifest.update({'migrationId': uuid.uuid4().hex, 'archiveFiles': files, 'pathBindings': bindings,
'archiveSha256': sha256(archive), 'nativeHistoryFiles': native_files,
'nativeHistorySha256': sha256(payload / 'native-history.tar.gz'),
'sourceProjectRoot': str(ROOT), 'modelProfilesMigrated': len((filtered or {}).get('profiles', [])),
'modelProfilesExcluded': len(excluded), 'modelConfigSha256': sha256(payload / 'model-config.json') if filtered else None,
'expiredMissingDrafts': missing_optional,
'sourceDatabaseBytes': int(backup.client("SELECT SUM(data_length+index_length) FROM information_schema.tables WHERE table_schema='" + database + "';").strip()),
'nativeHistoryDatabases': len(runtime_roots)})
private_json(work / 'migration-file-snapshot.json', manifest)
print('Model profiles migrated:', manifest['modelProfilesMigrated'], '; MiMo excluded:', len(excluded), flush=True)
print('Native business-history databases:', len(runtime_roots), '; files:', len(native_files), flush=True)
if __name__ == '__main__':
main()
+288
View File
@@ -0,0 +1,288 @@
#!/usr/bin/env python3
"""Build a signed, program-only Windows update bundle without network access."""
from __future__ import annotations
import argparse
import base64
from dataclasses import dataclass
import hashlib
import os
from pathlib import Path
import re
import stat
import tempfile
import zipfile
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey, Ed25519PublicKey
import update_manager
VERSION = re.compile(r'[A-Za-z0-9][A-Za-z0-9._-]{0,79}')
ARCHIVE_VERSION = re.compile(r'[A-Za-z0-9][A-Za-z0-9._-]{0,100}')
REQUIRED_PROGRAM_FILES = {
'app/backend.jar',
'deploy/offline/control.py',
'deploy/offline/update_manager.py',
'backend/src/main/resources/analysis-runtime-policy.json',
}
SENSITIVE_PARTS = {'.runtime', 'runtimedata', 'migration', 'private', 'secrets', 'credentials'}
SENSITIVE_SUFFIXES = {'.pem', '.key', '.p12', '.pfx', '.jks'}
class PublishError(RuntimeError):
"""Release inputs cannot produce a safe update package."""
@dataclass(frozen=True)
class PublishedArtifacts:
archive: Path
latest: Path
file_count: int
archive_sha256: str
def _manifest(root: Path) -> dict:
try:
value = update_manager.read(root / 'manifest.json')
except Exception as error:
raise PublishError('无法读取部署包 manifest.json。') from error
if not isinstance(value, dict) or value.get('platform') != 'windows' or value.get('architecture') != 'x86_64':
raise PublishError('输入目录必须是 Windows x64 离线部署包。')
if not isinstance(value.get('files'), list) or len(value['files']) > 50000:
raise PublishError('部署包文件清单无效。')
for key in ('runtimes', 'pythonWheels'):
if not isinstance(value.get(key), list):
raise PublishError('部署包缺少运行环境指纹信息。')
current_sequence = value.get('updateSequence', 0)
if type(current_sequence) is not int or current_sequence < 0:
raise PublishError('部署包更新序号无效。')
return value
def _sensitive_path(name: str) -> bool:
path = update_manager.safe_path(name)
parts = [part.casefold() for part in path.parts]
return (name == update_manager.KEY
or name.startswith('deploy/bootstrap/')
or any(part in SENSITIVE_PARTS for part in parts)
or path.suffix.casefold() in SENSITIVE_SUFFIXES)
def _program_inventory(root: Path, manifest: dict) -> list[dict]:
entries: dict[str, dict] = {}
folded: set[str] = set()
for entry in manifest['files']:
if not isinstance(entry, dict):
raise PublishError('部署包文件清单无效。')
name = entry.get('path')
try:
update_manager.safe_path(name)
except update_manager.UpdateError as error:
raise PublishError('部署包文件清单含有无效路径。') from error
if name.casefold() in folded:
raise PublishError('部署包文件清单含有重复或大小写冲突路径。')
folded.add(name.casefold())
if not update_manager.program_path(name):
continue
if _sensitive_path(name):
raise PublishError('程序清单中包含受保护的私有、迁移、引导或密钥文件路径。')
if entry.get('symlink') is not None or not update_manager.SHA.fullmatch(str(entry.get('sha256', ''))):
raise PublishError('程序文件清单缺少有效摘要或包含符号链接。')
path = update_manager.destination(root, name)
if not path.is_file() or path.is_symlink():
raise PublishError('程序清单文件缺失或不是普通文件。')
try:
digest = update_manager.digest(path)
except OSError as error:
raise PublishError('读取程序清单文件失败。') from error
if digest != entry['sha256']:
raise PublishError('部署包程序文件与 manifest.json 摘要不一致。')
entries[name] = {'path': name, 'sha256': digest, 'size': path.stat().st_size}
if not REQUIRED_PROGRAM_FILES.issubset(entries) or not any(name.startswith('app/web/') for name in entries):
raise PublishError('程序清单缺少更新所需的后端、网页、策略或更新管理程序。')
# The updater replaces program files as a complete snapshot. Refuse to
# sign an incomplete manifest if a program file exists outside its list.
present: set[str] = set()
for path in root.rglob('*'):
name = path.relative_to(root).as_posix()
if not update_manager.program_path(name):
continue
if path.is_symlink():
raise PublishError('程序目录含有符号链接或特殊文件。')
if path.is_dir():
continue
if not path.is_file():
raise PublishError('程序目录含有符号链接或特殊文件。')
present.add(name)
if present != set(entries):
raise PublishError('manifest.json 与磁盘上的完整程序文件清单不一致。')
return [entries[name] for name in sorted(entries)]
def _load_private_key(path: Path, root: Path, output_dir: Path) -> Ed25519PrivateKey:
try:
resolved = path.expanduser().resolve(strict=True)
if resolved.is_relative_to(root) or resolved.is_relative_to(output_dir):
raise PublishError('签名私钥必须保存在部署包和更新产物目录之外。')
key = serialization.load_pem_private_key(resolved.read_bytes(), password=None)
except PublishError:
raise
except Exception as error:
raise PublishError('无法读取未加密 PEM 格式的 Ed25519 签名私钥。') from error
if not isinstance(key, Ed25519PrivateKey):
raise PublishError('签名私钥必须是 Ed25519。')
return key
def _write_archive(root: Path, archive_temp: Path, files: list[dict]) -> None:
with zipfile.ZipFile(archive_temp, 'w', compression=zipfile.ZIP_DEFLATED, compresslevel=9, allowZip64=True) as output:
for entry in files:
name = entry['path']
source = update_manager.destination(root, name)
info = zipfile.ZipInfo(name, date_time=(1980, 1, 1, 0, 0, 0))
info.compress_type = zipfile.ZIP_DEFLATED
info.create_system = 3
info.external_attr = (stat.S_IFREG | 0o644) << 16
digest = hashlib.sha256()
size = 0
with source.open('rb') as reader, output.open(info, 'w') as writer:
while block := reader.read(1024 * 1024):
size += len(block)
if size > entry['size'] or size > update_manager.MAX_EXPANDED:
raise PublishError('程序文件在打包期间发生变化。')
digest.update(block)
writer.write(block)
if size != entry['size'] or digest.hexdigest() != entry['sha256']:
raise PublishError('程序文件在打包期间发生变化,已停止发布。')
def publish(root: Path, version: str, sequence: int, private_key_path: Path, output_dir: Path,
allow_baseline: bool = False) -> PublishedArtifacts:
"""Create quality-update-<version>.zip and signed latest.json locally."""
root = root.expanduser().resolve(strict=True)
if not root.is_dir():
raise PublishError('部署包根路径必须是目录。')
if not isinstance(version, str) or not VERSION.fullmatch(version):
raise PublishError('版本号只能包含字母、数字、点、下划线和连字符。')
if not ARCHIVE_VERSION.fullmatch(version):
raise PublishError('版本号不能用于更新文件名。')
if type(sequence) is not int or sequence <= 0:
raise PublishError('更新序号必须是正整数。')
manifest = _manifest(root)
current_sequence = manifest['updateSequence']
if sequence < current_sequence:
raise PublishError('更新序号不能低于部署包当前序号。')
if sequence == current_sequence and not allow_baseline:
raise PublishError('更新序号必须大于部署包当前序号。')
if sequence == current_sequence and version != manifest.get('updateVersion'):
raise PublishError('基线版本必须与部署包当前 updateVersion 完全一致。')
files = _program_inventory(root, manifest)
if sum(entry['size'] for entry in files) > update_manager.MAX_EXPANDED:
raise PublishError('更新程序文件总大小超过限制。')
output_dir = output_dir.expanduser().resolve()
if output_dir.is_relative_to(root):
raise PublishError('更新产物目录必须位于部署包目录之外。')
private_key = _load_private_key(Path(private_key_path), root, output_dir)
public_key_path = update_manager.destination(root, update_manager.KEY)
try:
public_key = serialization.load_pem_public_key(public_key_path.read_bytes())
except Exception as error:
raise PublishError('无法读取部署包中的更新公钥。') from error
if not isinstance(public_key, Ed25519PublicKey) or public_key.public_bytes(
serialization.Encoding.Raw, serialization.PublicFormat.Raw) != private_key.public_key().public_bytes(
serialization.Encoding.Raw, serialization.PublicFormat.Raw):
raise PublishError('签名私钥与部署包中的更新公钥不匹配。')
output_dir.mkdir(parents=True, exist_ok=True)
archive_name = f'quality-update-{version}.zip'
archive_path = output_dir / archive_name
latest_path = output_dir / 'latest.json'
if archive_path.exists() or latest_path.exists():
raise PublishError('输出目录已包含同名发布文件;请使用新的空输出目录。')
archive_temp: Path | None = None
latest_temp: Path | None = None
published_archive = False
try:
with tempfile.NamedTemporaryFile(prefix='.quality-update-', suffix='.tmp', dir=output_dir, delete=False) as stream:
archive_temp = Path(stream.name)
_write_archive(root, archive_temp, files)
archive_size = archive_temp.stat().st_size
if not 0 < archive_size <= update_manager.MAX_ARCHIVE:
raise PublishError('更新压缩包大小超过限制。')
archive_hash = update_manager.digest(archive_temp)
release = {
'schema': 1,
'version': version,
'sequence': sequence,
'platform': 'windows-x64',
'runtimeFingerprint': update_manager.runtime_fingerprint(manifest),
'archive': {'file': archive_name, 'size': archive_size, 'sha256': archive_hash},
'files': files,
}
raw = update_manager.canonical(release)
if len(raw) > update_manager.MAX_MANIFEST:
raise PublishError('更新文件清单超过大小限制。')
envelope = {
'payload': base64.b64encode(raw).decode('ascii'),
'signature': base64.b64encode(private_key.sign(raw)).decode('ascii'),
}
try:
update_manager.validate_release(envelope, root)
except update_manager.UpdateError as error:
raise PublishError('生成的更新包未通过客户端契约校验:' + str(error)) from error
latest_bytes = update_manager.canonical(envelope)
with tempfile.NamedTemporaryFile(prefix='.latest-', suffix='.tmp', dir=output_dir, delete=False) as stream:
latest_temp = Path(stream.name)
stream.write(latest_bytes)
stream.flush()
os.fsync(stream.fileno())
os.link(archive_temp, archive_path)
archive_temp.unlink()
archive_temp = None
published_archive = True
os.link(latest_temp, latest_path)
latest_temp.unlink()
latest_temp = None
return PublishedArtifacts(archive_path, latest_path, len(files), archive_hash)
except FileExistsError as error:
raise PublishError('输出目录出现同名发布文件;请使用新的空输出目录。') from error
except OSError as error:
raise PublishError('写入更新产物失败。') from error
finally:
if archive_temp is not None:
archive_temp.unlink(missing_ok=True)
if latest_temp is not None:
latest_temp.unlink(missing_ok=True)
if published_archive and not latest_path.exists():
archive_path.unlink(missing_ok=True)
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--root', type=Path, required=True, help='Windows x64 offline bundle directory')
parser.add_argument('--version', required=True, help='Release version, for example 2026.11.04.2')
parser.add_argument('--sequence', type=int, required=True, help='Monotonically increasing update sequence')
parser.add_argument('--private-key', type=Path, required=True, help='Ed25519 PEM private key kept outside the bundle')
parser.add_argument('--output-dir', type=Path, required=True, help='New output directory for the ZIP and latest.json')
parser.add_argument('--allow-baseline', action='store_true',
help='Allow a seed release only when version and sequence exactly match the bundle manifest')
args = parser.parse_args(argv)
try:
result = publish(args.root, args.version, args.sequence, args.private_key, args.output_dir, args.allow_baseline)
except (PublishError, update_manager.UpdateError) as error:
parser.error(str(error))
print(f'已生成更新包:{result.archive}')
print(f'已生成签名清单:{result.latest}')
print(f'版本 {args.version},序号 {args.sequence},程序文件 {result.file_count} 个,SHA-256 {result.archive_sha256}')
return 0
if __name__ == '__main__':
raise SystemExit(main())
+327
View File
@@ -0,0 +1,327 @@
"""Restore into a new database, validate, then switch the Windows deployment."""
from __future__ import annotations
from datetime import datetime, timezone
import gzip
import json
import os
from pathlib import Path, PurePosixPath
import re
import shutil
import sqlite3
import subprocess
import sys
import time
import uuid
import control
from migration_common import extract_checked, query_evidence, relative_name, sha256, sql_text, table_counts
ROOT = control.ROOT
PAYLOAD = ROOT / 'migration'
JOURNAL = control.PRIVATE / 'migration-progress.json'
RECEIPT = control.PRIVATE / 'migration-receipt.json'
def checked_local(relative):
rel = relative_name(relative)
path = ROOT.joinpath(*rel.parts)
if not path.resolve().is_relative_to(ROOT.resolve()):
raise RuntimeError('恢复暂存路径越界。')
return path
def run_query(value, statement, database=None):
command = control.mysql_command(value)
command += ['--batch', '--skip-column-names', '--default-character-set=utf8mb4', '--max-allowed-packet=1G']
if database:
if not re.fullmatch(r'quality_(?:platform|restore_[a-f0-9]{12,32})', database):
raise RuntimeError('恢复数据库名称无效。')
command += ['--database=' + database]
env = control.environment(value)
env['MYSQL_PWD'] = value['rootPassword']
result = subprocess.run(command, input=statement, text=True, encoding='utf-8',
stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env, timeout=1800)
if result.returncode:
raise RuntimeError('恢复数据库校验未完成;已有数据库保留。')
return result.stdout
def stop_owned_service():
state = control.read_state()
if control.matches(state.get('supervisor', {})):
control.atomic(control.STOP, {'token': state['token']})
for _ in range(300):
if not control.matches(state['supervisor']):
break
time.sleep(.5)
else:
raise RuntimeError('原服务仍在关闭,请稍后重新运行。')
if any(control.matches(info) for info in state.get('children', [])):
raise RuntimeError('发现本目录尚未关闭的后台进程,请检查服务日志。')
def start_database(value):
with (control.DATA / 'logs/migration-database-supervisor.log').open('ab') as log:
options = ({'creationflags': subprocess.DETACHED_PROCESS | subprocess.CREATE_NEW_PROCESS_GROUP}
if control.IS_WINDOWS else {'start_new_session': True})
process = subprocess.Popen([str(control.PYTHON), str(Path(control.__file__).resolve()),
'serve', '--database-only'], cwd=ROOT, env=control.environment(value),
stdin=subprocess.DEVNULL, stdout=log, stderr=log, **options)
for _ in range(360):
state = control.read_state()
if (state.get('ready') and state.get('databaseOnly') and
state.get('supervisor', {}).get('pid') == process.pid and control.matches(state['supervisor'])):
return
if process.poll() is not None:
raise RuntimeError('恢复数据库无法启动,请查看 runtimedata/logs。')
time.sleep(.5)
raise RuntimeError('恢复数据库仍未就绪,请查看日志后重试。')
def restore_dump(value, database, dump):
env = control.environment(value)
env['MYSQL_PWD'] = value['rootPassword']
command = control.mysql_command(value) + ['--database=' + database,
'--default-character-set=utf8mb4', '--max-allowed-packet=1G',
'--init-command=SET SESSION autocommit=0']
with (control.DATA / 'logs/migration-import.log').open('ab') as errors:
process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=errors, env=env)
try:
total, next_progress = dump.stat().st_size, 0
with dump.open('rb') as raw, gzip.GzipFile(fileobj=raw) as source:
while chunk := source.read(1024 * 1024):
process.stdin.write(chunk)
percent = min(99, raw.tell() * 100 // total)
if percent >= next_progress:
print(f'数据库导入进度约 {percent}%……', flush=True)
next_progress = percent + 10
process.stdin.write(b'\nCOMMIT;\n')
process.stdin.close()
if process.wait(timeout=1800):
raise RuntimeError('业务数据导入失败,请查看私有恢复日志;原数据库未覆盖。')
except BaseException:
if process.poll() is None:
process.kill()
process.wait()
raise
def rebind_sources(query, manifest, archive):
counts = {'source_revision': 0, 'source_candidate': 0, 'source_upload_draft': 0}
for entry in manifest['pathBindings']:
pair = (entry['table'], entry['column'])
if pair not in {('source_revision', 'archive_path'), ('source_candidate', 'source_path'),
('source_candidate', 'archive_path'), ('source_upload_draft', 'source_path')}:
raise RuntimeError('来源路径绑定类型无效。')
rel = relative_name(entry['relative'])
target = archive.joinpath(*rel.parts)
if not entry['expiredMissing']:
if not target.is_file() or target.stat().st_size != entry['bytes'] or sha256(target) != entry['sha256']:
raise RuntimeError('来源文件校验失败,恢复未切换。')
query('UPDATE `' + entry['table'] + '` SET `' + entry['column'] + '`=' + sql_text(str(target))
+ ' WHERE id=' + sql_text(entry['id']) + ' AND `' + entry['column'] + '`=' + sql_text(entry['oldPath']) + ';')
actual = query('SELECT `' + entry['column'] + '` FROM `' + entry['table'] + '` WHERE id=' + sql_text(entry['id']) + ';').strip()
if actual != str(target):
raise RuntimeError('来源文件路径重绑定失败。')
counts[entry['table']] += 1
return counts
def relocate_native(stage, source_root):
source_root = PurePosixPath(source_root)
def relocated(value):
if isinstance(value, str) and value.startswith(str(source_root) + '/'):
relative = PurePosixPath(value).relative_to(source_root)
if '..' in relative.parts:
raise RuntimeError('原生历史路径无效。')
return str(ROOT.joinpath(*relative.parts))
if isinstance(value, dict):
return {key: relocated(child) for key, child in value.items()}
if isinstance(value, list):
return [relocated(child) for child in value]
return value
for path in stage.rglob('*.json'):
control.atomic(path, relocated(json.loads(path.read_text(encoding='utf-8'))))
databases = 0
for path in stage.rglob('nao.sqlite'):
connection = sqlite3.connect(path)
try:
connection.execute('PRAGMA secure_delete=ON')
for table, column in (('project', 'path'), ('message_part', 'storage_path')):
exists = connection.execute("SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (table,)).fetchone()
if not exists:
continue
for rowid, original in connection.execute('SELECT rowid,"' + column + '" FROM "' + table + '" WHERE "' + column + '" IS NOT NULL').fetchall():
replacement = relocated(original)
if replacement != original:
connection.execute('UPDATE "' + table + '" SET "' + column + '"=? WHERE rowid=?', (replacement, rowid))
connection.commit()
connection.execute('VACUUM')
if connection.execute('PRAGMA integrity_check').fetchone()[0] != 'ok':
raise RuntimeError('原生分析历史数据库校验失败。')
databases += 1
finally:
connection.close()
return databases
def rollback_plan(plan):
# Only paths generated by this migration are eligible. Databases and source
# archive trees are retained even after an interrupted restore.
attempt = plan['attempt']
if not re.fullmatch(r'[a-f0-9]{16}', attempt):
raise RuntimeError('恢复日志无效。')
for item in plan['privateSwitches']:
target, previous, staged = (checked_local(item[key]) for key in ('target', 'previous', 'staged'))
failed = checked_local(item['failed'])
if previous.exists():
if target.exists():
target.rename(failed)
previous.rename(target)
elif not item['existed'] and not staged.exists() and target.exists():
target.rename(failed)
control.atomic(control.CONFIG, plan['previousConfig'])
JOURNAL.rename(control.PRIVATE / ('migration-rolled-back-' + attempt + '.json'))
def commit_plan(plan, receipt):
control.atomic(JOURNAL, plan)
try:
for item in plan['privateSwitches']:
target, previous, staged = (checked_local(item[key]) for key in ('target', 'previous', 'staged'))
if target.is_symlink() or staged.is_symlink() or previous.exists():
raise RuntimeError('私有配置切换路径无效。')
if target.exists():
target.rename(previous)
staged.rename(target)
control.atomic(control.CONFIG, plan['newConfig'])
control.atomic(RECEIPT, receipt)
except BaseException:
rollback_plan(plan)
raise
JOURNAL.unlink(missing_ok=True)
def restore(check_runtime=True):
control.private_directory(control.PRIVATE)
control.private_directory(control.DATA)
(control.DATA / 'logs').mkdir(exist_ok=True)
(control.DATA / 'sources').mkdir(exist_ok=True)
(control.DATA / 'archive').mkdir(exist_ok=True)
lock = os.open(control.PRIVATE / 'migration.lock', os.O_CREAT | os.O_RDWR, 0o600)
try:
control.host.fcntl.flock(lock, control.host.fcntl.LOCK_EX | control.host.fcntl.LOCK_NB)
except BlockingIOError:
os.close(lock)
raise RuntimeError('另一项迁移正在进行,请等待原窗口完成。') from None
try:
return restore_locked(check_runtime)
finally:
os.close(lock)
def restore_locked(check_runtime):
manifest_path = PAYLOAD / 'manifest.json'
manifest = json.loads(manifest_path.read_text(encoding='utf-8'))
if manifest.get('formatVersion') != 1 or not re.fullmatch(r'[a-f0-9]{32}', manifest.get('migrationId', '')):
raise RuntimeError('迁移包格式无效。')
migration_id = manifest['migrationId']
if RECEIPT.exists():
receipt = json.loads(RECEIPT.read_text(encoding='utf-8'))
current = control.configuration()
if receipt.get('migrationId') == migration_id and current.get('restoredFromMigration') == migration_id and current.get('mysqlDatabase') == receipt.get('database'):
print('此迁移已完成,保留 Windows 上的后续数据,直接启动。', flush=True)
JOURNAL.unlink(missing_ok=True)
return receipt
raise RuntimeError('此目录已有其他迁移记录,请使用新的部署目录。')
stop_owned_service()
if JOURNAL.exists():
rollback_plan(json.loads(JOURNAL.read_text(encoding='utf-8')))
print('上次未完成的切换已撤回,原配置已保留。', flush=True)
value = control.verify() if check_runtime else control.configuration()
print('正在校验迁移包……', flush=True)
for name, key in (('database.sql.gz', 'databaseDumpSha256'), ('archive.tar.gz', 'archiveSha256'),
('native-history.tar.gz', 'nativeHistorySha256')):
if sha256(PAYLOAD / name) != manifest[key]:
raise RuntimeError('迁移包文件不完整,已停止恢复。')
if manifest.get('modelConfigSha256') and sha256(PAYLOAD / 'model-config.json') != manifest['modelConfigSha256']:
raise RuntimeError('模型配置校验失败。')
if shutil.disk_usage(ROOT).free < manifest['minimumFreeBytes']:
raise RuntimeError('可用磁盘空间不足,请至少预留 ' + str((manifest['minimumFreeBytes'] + 1024**3 - 1) // 1024**3) + ' GB。')
attempt = uuid.uuid4().hex[:16]
stage = control.PRIVATE / ('migration-stage-' + attempt)
stage.mkdir(mode=0o700)
archive = control.DATA / 'migrations' / attempt / 'archive'
archive.parent.mkdir(parents=True, mode=0o700)
native = stage / 'nao-users'
print('正在恢复来源文件和分析历史……', flush=True)
extract_checked(PAYLOAD / 'archive.tar.gz', archive, manifest['archiveFiles'])
extract_checked(PAYLOAD / 'native-history.tar.gz', native, manifest['nativeHistoryFiles'])
native_count = relocate_native(native, manifest['sourceProjectRoot'])
if native_count != manifest['nativeHistoryDatabases']:
raise RuntimeError('分析历史文件数量不一致。')
model = stage / 'model-config.json'
if manifest.get('modelConfigSha256'):
shutil.copy2(PAYLOAD / 'model-config.json', model)
model.chmod(0o600)
settings = json.loads(model.read_text(encoding='utf-8'))
if any('mimo' in str(profile.get(key, '')).lower() for profile in settings['profiles'] for key in ('model', 'name', 'baseUrl')):
raise RuntimeError('迁移模型配置中仍含 MiMo。')
database = 'quality_restore_' + uuid.uuid4().hex
try:
start_database(value)
run_query(value, 'CREATE DATABASE `' + database + '` CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;')
restore_dump(value, database, PAYLOAD / 'database.sql.gz')
query = lambda statement: run_query(value, statement, database)
print('正在核对全部数据表、原账号和正式统计结果……', flush=True)
actual_counts = table_counts(query)
if actual_counts != manifest['tableCounts']:
raise RuntimeError('恢复后的数据表行数不一致,未切换正式使用库。')
actual_users = [json.loads(line) for line in query("SELECT JSON_OBJECT('id',id,'username',username,'passwordHashSha256',SHA2(password_hash,256),'active',active) FROM app_user ORDER BY id;").splitlines()]
if sorted(actual_users, key=lambda row: row['id']) != sorted(manifest['sourceUsers'], key=lambda row: row['id']):
raise RuntimeError('原用户账号校验失败,未切换使用库。')
if query_evidence(query) != manifest['queryEvidence']:
raise RuntimeError('正式版本和月度统计结果不一致,未切换使用库。')
rebound = rebind_sources(query, manifest, archive)
run_query(value, 'GRANT ALL PRIVILEGES ON `' + database + "`.* TO 'quality_app'@'127.0.0.1';")
run_query(value, 'FLUSH TABLES;')
finally:
stop_owned_service()
new_config = {**value, 'mysqlDatabase': database, 'archiveDirectory': archive.relative_to(ROOT).as_posix(),
'restoredFromMigration': migration_id, 'webBind': '0.0.0.0'}
for name in ('dataWorkflowTestMode', 'analysisAutonomousEnabled', 'analysisProtocolV1Enabled'):
if name in manifest.get('applicationSettings', {}):
new_config[name] = bool(manifest['applicationSettings'][name])
switches = []
for staged, target in [(native, control.PRIVATE / 'nao-users')] + ([(model, control.PRIVATE / 'nao-model-config.json')] if model.exists() else []):
switches.append({'staged': staged.relative_to(ROOT).as_posix(), 'target': target.relative_to(ROOT).as_posix(),
'previous': (control.PRIVATE / ('before-migration-' + attempt + '-' + target.name)).relative_to(ROOT).as_posix(),
'failed': (control.PRIVATE / ('failed-migration-' + attempt + '-' + target.name)).relative_to(ROOT).as_posix(),
'existed': target.exists()})
receipt = {'migrationId': migration_id, 'migrationManifestSha256': sha256(manifest_path), 'database': database,
'restoredAtUtc': datetime.now(timezone.utc).isoformat(), 'tables': len(actual_counts),
'totalRows': sum(actual_counts.values()), 'sourceReferencesRebound': rebound,
'modelProfilesMigrated': manifest['modelProfilesMigrated'], 'modelProfilesExcluded': manifest['modelProfilesExcluded'],
'nativeHistoryDatabases': native_count, 'originalDatabasesPreserved': True,
'archiveDirectory': new_config['archiveDirectory']}
plan = {'attempt': attempt, 'previousConfig': value, 'newConfig': new_config, 'privateSwitches': switches}
commit_plan(plan, receipt)
print('迁移恢复完成。请使用原来的账号和密码登录;MiMo 配置未迁移。', flush=True)
return receipt
if __name__ == '__main__':
try:
restore()
except (RuntimeError, OSError, ValueError, subprocess.SubprocessError) as error:
import traceback
log = control.DATA / 'logs/migration-validation.log'
log.parent.mkdir(parents=True, exist_ok=True)
fd = os.open(log, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o600)
with os.fdopen(fd, 'w', encoding='utf-8') as output:
traceback.print_exc(file=output)
if isinstance(error, RuntimeError):
print(str(error), file=sys.stderr)
print('迁移未完成。已有数据库保留;原配置在切换失败时自动恢复。请查看私有恢复日志后重试。', file=sys.stderr)
raise SystemExit(1)
+87
View File
@@ -0,0 +1,87 @@
"""Platform helpers for the standalone distribution; POSIX behavior is preserved."""
from __future__ import annotations
import errno
import os
import signal
import time
IS_WINDOWS = os.name == 'nt'
if os.name != 'nt':
import fcntl
else:
import msvcrt
class _FileLocks:
LOCK_EX, LOCK_NB, LOCK_UN = 2, 4, 8
@staticmethod
def flock(fd, operation):
os.lseek(fd, 0, os.SEEK_SET)
if operation & _FileLocks.LOCK_UN:
msvcrt.locking(fd, msvcrt.LK_UNLCK, 1)
return
while True:
try:
msvcrt.locking(fd, msvcrt.LK_NBLCK, 1)
return
except OSError as error:
if error.errno not in (errno.EACCES, errno.EAGAIN, errno.EDEADLK):
raise
if operation & _FileLocks.LOCK_NB:
raise BlockingIOError(errno.EAGAIN, 'File is locked') from None
time.sleep(.05)
fcntl = _FileLocks()
def private_bits(mode):
# NTFS access is protected by the inherited user-only ACL on .runtime.
# Windows' st_mode has no information about that ACL.
return not IS_WINDOWS and bool(mode & 0o077)
def owner_mismatch(info):
return not IS_WINDOWS and info.st_uid != os.getuid()
def fchmod(fd, mode):
if os.name != 'nt':
os.fchmod(fd, mode)
def ps_value(pid, field):
import psutil
try:
process = psutil.Process(int(pid))
if not process.is_running() or process.status() == psutil.STATUS_ZOMBIE:
return None
if field == 'lstart':
return repr(process.create_time())
if field == 'command':
return ' '.join(process.cmdline())
return None
except (psutil.Error, ValueError):
return None
def group_leader(pid):
return True if os.name == 'nt' else os.getpgid(pid) == pid
def killpg(pid, sig):
if os.name != 'nt':
return os.killpg(pid, sig)
import psutil
try:
process = psutil.Process(pid)
# Caller verifies creation time and executable before reaching this point.
children = process.children(recursive=True)
for child in reversed(children):
try:
child.kill() if sig == getattr(signal, 'SIGKILL', 9) else child.terminate()
except psutil.NoSuchProcess:
pass
process.kill() if sig == getattr(signal, 'SIGKILL', 9) else process.terminate()
except psutil.NoSuchProcess:
raise ProcessLookupError(pid) from None
+117
View File
@@ -0,0 +1,117 @@
"""Exercise an isolated empty MySQL database with the host's native test runtimes."""
import argparse
import http.cookiejar
import json
import os
from pathlib import Path
import secrets
import shutil
import socket
import subprocess
import sys
import tempfile
import time
import urllib.request
from build_bundle import ROOT, TEMPLATES, adapt_runtime
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--work', type=Path, required=True)
args = parser.parse_args()
work = args.work.resolve()
stage = Path(tempfile.mkdtemp(prefix='qc-offline-', dir='/tmp')).resolve()
(work / 'smoke-root.txt').write_text(str(stage))
for folder in ('deploy/offline', 'scripts', 'runtimes', 'app', '.runtime'):
(stage / folder).mkdir(parents=True, exist_ok=True, mode=0o700)
for name in ('control.py', 'gateway.py'):
shutil.copy2(TEMPLATES / name, stage / 'deploy/offline' / name)
shutil.copy2(TEMPLATES / 'runtime_platform.py', stage / 'scripts/runtime_platform.py')
for pattern in ('nao_*.py', 'analysis_visual*.py'):
for path in (ROOT / 'scripts').glob(pattern):
(stage / 'scripts' / path.name).write_text(adapt_runtime(path.read_text()) if path.name == 'nao_quality_runtime.py' else path.read_text())
shutil.copytree(ROOT / 'ai/nao-quality', stage / 'ai/nao-quality')
shutil.copy2(next((work / 'source/backend/target').glob('*.jar')), stage / 'app/backend.jar')
shutil.copytree(work / 'source/web/dist', stage / 'app/web')
for name, target in (('java', ROOT / '.runtime/jdk/Contents/Home'), ('mysql', ROOT / '.runtime/mysql'),
('python', ROOT / '.runtime/nao-trial/venv')):
(stage / 'runtimes' / name).symlink_to(target, target_is_directory=True)
def available(port):
try:
with socket.socket() as test:
test.bind(('127.0.0.1', port))
return True
except OSError:
return False
port = next(p for p in (5284, 5283, 5285) if available(p))
def free():
with socket.socket() as test:
test.bind(('127.0.0.1', 0))
return test.getsockname()[1]
value = {'mysqlPort': free(), 'backendPort': port, 'webPort': free(), 'webBind': '127.0.0.1',
'mysqlPassword': secrets.token_urlsafe(32), 'rootPassword': secrets.token_urlsafe(32),
'adminUsername': 'admin', 'adminPassword': secrets.token_urlsafe(24)}
config = stage / '.runtime/deployment.json'
config.write_text(json.dumps(value))
config.chmod(0o600)
report = {'environment': 'macOS isolated empty database using host-native runtimes',
'targetWindowsLinuxNativeValidated': False, 'checks': []}
process = None
try:
for cycle in range(2):
with (stage / f'cycle-{cycle}.log').open('w') as log:
process = subprocess.Popen([sys.executable, str(stage / 'deploy/offline/control.py'), 'serve'], stdout=log, stderr=subprocess.STDOUT)
for _ in range(180):
if process.poll() is not None:
raise RuntimeError('Supervisor exited; inspect ' + str(stage / f'cycle-{cycle}.log'))
try:
state = json.loads((stage / '.runtime/service.json').read_text())
if state.get('ready'):
break
except (FileNotFoundError, ValueError):
pass
time.sleep(1)
else:
raise RuntimeError('Isolated startup timed out')
cookies = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}), urllib.request.HTTPCookieProcessor(cookies))
base = f'http://127.0.0.1:{value["webPort"]}'
with opener.open(base + '/', timeout=5) as response:
assert response.status == 200 and b'<div id="root">' in response.read()
with opener.open(base + '/actuator/health', timeout=5) as response:
assert json.load(response)['status'] == 'UP'
with opener.open(base + '/api/auth/csrf', timeout=5) as response:
token = json.load(response)['token']
request = urllib.request.Request(base + '/api/auth/login',
data=json.dumps({'username': value['adminUsername'], 'password': value['adminPassword']}).encode(),
headers={'Content-Type': 'application/json', 'X-XSRF-TOKEN': token})
with opener.open(request, timeout=10) as response:
assert response.status == 200
for path in ('/api/auth/me', '/api/options'):
with opener.open(base + path, timeout=10) as response:
assert response.status == 200
report['checks'].append({'cycle': cycle + 1, 'web': True, 'databaseMigration': True,
'csrfAndLogin': True, 'authenticatedOptions': True})
result = subprocess.run([sys.executable, str(stage / 'deploy/offline/control.py'), 'stop'], capture_output=True, text=True, timeout=150)
if result.returncode:
raise RuntimeError('Stop failed: ' + result.stderr)
process.wait(timeout=10)
process = None
assert not (stage / '.runtime/service.json').exists()
print('Isolated startup/login/stop cycle', cycle + 1, 'passed', flush=True)
report['passed'] = True
except Exception as error:
report['passed'] = False
report['error'] = str(error)
print(str(error), flush=True)
finally:
if process is not None and process.poll() is None:
subprocess.run([sys.executable, str(stage / 'deploy/offline/control.py'), 'stop'], capture_output=True, timeout=150)
(work / 'smoke-report.json').write_text(json.dumps(report, ensure_ascii=False, indent=2) + '\n')
print(json.dumps(report, ensure_ascii=False), flush=True)
return 0 if report['passed'] else 1
if __name__ == '__main__':
raise SystemExit(main())
+148
View File
@@ -0,0 +1,148 @@
import gzip
import hashlib
import io
import json
import os
from pathlib import Path
import sqlite3
import sys
import tarfile
import tempfile
import unittest
from unittest.mock import patch
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE))
import control
import restore_migration as restore
from migration_common import dump_counts, extract_checked, filter_models, inserted_rows, path_under_source, relative_name
from prepare_migration import native_copy
class MigrationValidationTest(unittest.TestCase):
def test_extended_insert_counts_ignore_strings_and_escaped_quotes(self):
cases = [
(b"INSERT INTO `x` VALUES (1,'a'),(2,'b');\n", 2),
(b"INSERT INTO `x` VALUES (1,'fake ),( delimiter'),(2,'quote \\' ),( escaped'),(3,0x001122);\n", 3),
(b"INSERT INTO `x` VALUES ('a\\\\',NULL,'(nested)'),(NULL,'\\n',4);\n", 2),
(b"INSERT INTO `x` VALUES ('\xe4\xb8\xad\xe6\x96\x87', '{\\\"text\\\":\\\"),(\\\"}');\n", 1),
]
for statement, count in cases:
with self.subTest(statement=statement):
self.assertEqual(inserted_rows(statement), count)
def test_dump_counts_include_empty_tables(self):
with tempfile.TemporaryDirectory() as folder:
path = Path(folder) / 'dump.gz'
with gzip.open(path, 'wb') as stream:
stream.write(b"CREATE TABLE `empty` (id INT);\nCREATE TABLE `x` (id INT);\nINSERT INTO `x` VALUES (1),(2);\n")
counts, size = dump_counts(path)
self.assertEqual(counts, {'empty': 0, 'x': 2})
self.assertGreater(size, 0)
def test_mimo_profiles_and_root_selection_are_excluded(self):
config = {'revision': 'original', 'activeProfileId': 'mimo', 'profiles': [
{'id': 'mimo', 'name': 'MiMo', 'model': 'mimo-v2.6', 'apiKey': 'exclude-this', 'baseUrl': 'https://api.example.test'},
{'id': 'other', 'name': 'Qwen', 'model': 'qwen-test', 'apiKey': 'retain-this', 'baseUrl': 'https://other.example.test'}]}
filtered, removed = filter_models(config)
self.assertEqual(len(removed), 1)
self.assertEqual(filtered['activeProfileId'], 'other')
self.assertEqual(filtered['apiKey'], 'retain-this')
self.assertEqual(filtered['revision'], 'original')
self.assertNotIn('exclude-this', json.dumps(filtered))
self.assertEqual(len(filtered['profiles']), 1)
def test_mac_path_is_parsed_portably(self):
self.assertEqual(path_under_source('/Users/a/archive/source/文件.xlsx', '/Users/a/archive'), 'source/文件.xlsx')
for name in ('../outside', '/absolute', 'C:/data', 'a\\b', '.', 'a/./b', 'CON.txt', 'x/name.'):
with self.subTest(name=name), self.assertRaises(ValueError):
relative_name(name)
with self.assertRaises(ValueError):
path_under_source('/Users/a/archive-other/secret', '/Users/a/archive')
def test_archive_rejects_traversal_links_duplicates_and_bad_hash(self):
for kind in ('traversal', 'link', 'duplicate', 'checksum'):
with self.subTest(kind=kind), tempfile.TemporaryDirectory() as directory:
folder = Path(directory)
archive = folder / 'archive.gz'
body = b'contents'
with tarfile.open(archive, 'w:gz') as output:
info = tarfile.TarInfo('../outside' if kind == 'traversal' else 'file')
if kind == 'link':
info.type, info.linkname = tarfile.SYMTYPE, '/outside'
output.addfile(info)
else:
info.size = len(body)
output.addfile(info, io.BytesIO(body))
if kind == 'duplicate':
output.addfile(info, io.BytesIO(body))
expected = {'file': {'bytes': len(body), 'sha256': '0' * 64 if kind == 'checksum' else hashlib.sha256(body).hexdigest()}}
with self.assertRaises(ValueError):
extract_checked(archive, folder / 'target', expected)
self.assertFalse((folder / 'outside').exists())
def test_native_history_preserves_chats_and_scrubs_credentials(self):
with tempfile.TemporaryDirectory() as directory:
folder = Path(directory)
source = folder / 'source.sqlite'
db = sqlite3.connect(source)
db.executescript("CREATE TABLE chat(id TEXT,body TEXT);INSERT INTO chat VALUES('a','history');"
"CREATE TABLE project_llm_config(api_key TEXT);INSERT INTO project_llm_config VALUES('mimo-exclusive-secret');"
"CREATE TABLE session(token TEXT);INSERT INTO session VALUES('transient-token');")
db.commit()
db.close()
target = folder / 'target.sqlite'
native_copy(source, target)
migrated = sqlite3.connect(target)
self.assertEqual(migrated.execute('SELECT body FROM chat').fetchone()[0], 'history')
self.assertEqual(migrated.execute('SELECT COUNT(*) FROM project_llm_config').fetchone()[0], 0)
self.assertEqual(migrated.execute('SELECT COUNT(*) FROM session').fetchone()[0], 0)
migrated.close()
self.assertNotIn(b'mimo-exclusive-secret', target.read_bytes())
class MigrationCommitTest(unittest.TestCase):
def test_partial_file_switch_rolls_back_existing_files_and_configuration(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
private = root / '.runtime'
private.mkdir()
target = private / 'nao-users'
target.mkdir()
(target / 'old').write_text('original')
staged = private / 'stage-native'
staged.mkdir()
(staged / 'new').write_text('new')
model = private / 'nao-model-config.json'
model.write_text('original model')
attempt = 'a' * 16
plan = {'attempt': attempt, 'previousConfig': {'previous': True}, 'newConfig': {'new': True}, 'privateSwitches': [
{'target': '.runtime/nao-users', 'previous': '.runtime/previous-native', 'staged': '.runtime/stage-native', 'failed': '.runtime/failed-native', 'existed': True},
{'target': '.runtime/nao-model-config.json', 'previous': '.runtime/previous-model', 'staged': '.runtime/missing-stage-model', 'failed': '.runtime/failed-model', 'existed': True}]}
with patch.multiple(restore, ROOT=root, JOURNAL=private / 'journal', RECEIPT=private / 'receipt'), patch.object(control, 'CONFIG', private / 'deployment.json'):
with self.assertRaises(FileNotFoundError):
restore.commit_plan(plan, {})
self.assertEqual((target / 'old').read_text(), 'original')
self.assertEqual(model.read_text(), 'original model')
self.assertEqual(json.loads((private / 'deployment.json').read_text()), {'previous': True})
self.assertTrue((private / 'failed-native/new').exists())
def test_repeated_restore_does_not_reimport_old_snapshot(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
payload = root / 'migration'
payload.mkdir()
migration_id = 'a' * 32
(payload / 'manifest.json').write_text(json.dumps({'formatVersion': 1, 'migrationId': migration_id}))
receipt = root / 'receipt'
value = {'migrationId': migration_id, 'database': 'quality_restore_' + 'b' * 32}
receipt.write_text(json.dumps(value))
config = {'restoredFromMigration': migration_id, 'mysqlDatabase': value['database']}
with patch.multiple(restore, PAYLOAD=payload, RECEIPT=receipt, JOURNAL=root / 'journal'), patch.object(control, 'configuration', return_value=config), patch.object(restore, 'stop_owned_service') as stop, patch.object(restore, 'run_query') as query:
self.assertEqual(restore.restore_locked(False), value)
stop.assert_not_called()
query.assert_not_called()
if __name__ == '__main__':
unittest.main()
+179
View File
@@ -0,0 +1,179 @@
import functools
import http.client
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import importlib.util
import json
import os
from pathlib import Path
import socket
import subprocess
import sys
import tempfile
import threading
import time
import unittest
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE))
from gateway import make_server
from build_bundle import adapt_runtime
import runtime_platform
class Upstream(BaseHTTPRequestHandler):
def log_message(self, *_):
pass
def do_GET(self):
if self.path == '/api/events':
self.send_response(200)
self.send_header('Content-Type', 'text/event-stream')
self.end_headers()
self.wfile.write(b'data: first\n\n')
self.wfile.flush()
time.sleep(.8)
self.wfile.write(b'data: second\n\n')
self.wfile.flush()
else:
body = json.dumps({'cookie': self.headers.get('Cookie'), 'host': self.headers.get('Host')}).encode()
self.send_response(401)
self.send_header('Set-Cookie', 'one=1; HttpOnly')
self.send_header('Set-Cookie', 'two=2; SameSite=Strict')
self.send_header('Content-Length', str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_POST(self):
body = self.rfile.read(int(self.headers['Content-Length']))
self.send_response(201)
self.send_header('Content-Length', str(len(body)))
self.send_header('X-CSRF-Echo', self.headers.get('X-XSRF-TOKEN', ''))
self.end_headers()
self.wfile.write(body)
class GatewayTest(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.temp = tempfile.TemporaryDirectory()
root = Path(cls.temp.name)
(root / 'index.html').write_text('QUALITY APP', encoding='utf-8')
(root / 'assets').mkdir()
(root / 'assets/file.js').write_text('const x=1;')
cls.backend = ThreadingHTTPServer(('127.0.0.1', 0), Upstream)
cls.frontend = make_server(root, '127.0.0.1', 0, cls.backend.server_port)
for server in (cls.backend, cls.frontend):
threading.Thread(target=server.serve_forever, daemon=True).start()
@classmethod
def tearDownClass(cls):
for server in (cls.frontend, cls.backend):
server.shutdown()
server.server_close()
cls.temp.cleanup()
def connection(self):
return http.client.HTTPConnection('127.0.0.1', self.frontend.server_port, timeout=3)
def test_static_and_spa(self):
for url, body in [('/', b'QUALITY APP'), ('/?page=home', b'QUALITY APP'), ('/analysis', b'QUALITY APP'), ('/assets/file.js', b'const x=1;')]:
connection = self.connection()
connection.request('GET', url)
response = connection.getresponse()
self.assertEqual(response.status, 200)
self.assertEqual(response.read(), body)
connection.close()
def test_no_directory_listing_or_private_files(self):
for url in ('/assets/', '/.runtime/deployment.json', '/../.runtime/deployment.json', '/missing.js'):
connection = self.connection()
connection.request('GET', url)
response = connection.getresponse()
self.assertEqual(response.status, 404)
response.read()
connection.close()
def test_auth_cookies_and_host_are_preserved(self):
connection = self.connection()
connection.request('GET', '/api/auth/me', headers={'Cookie': 'SESSION=abc', 'Host': '192.168.1.22:5280'})
response = connection.getresponse()
self.assertEqual(response.status, 401)
self.assertEqual(len(response.headers.get_all('Set-Cookie')), 2)
self.assertEqual(json.loads(response.read()), {'cookie': 'SESSION=abc', 'host': '192.168.1.22:5280'})
connection.close()
def test_upload_and_csrf(self):
connection = self.connection()
data = os.urandom(2 * 1024 * 1024 + 19)
connection.request('POST', '/api/sources/upload', data, {'X-XSRF-TOKEN': 'csrf-value'})
response = connection.getresponse()
self.assertEqual(response.status, 201)
self.assertEqual(response.getheader('X-CSRF-Echo'), 'csrf-value')
self.assertEqual(response.read(), data)
connection.close()
def test_sse_is_streamed_before_upstream_finishes(self):
connection = self.connection()
started = time.monotonic()
connection.request('GET', '/api/events')
response = connection.getresponse()
self.assertEqual(response.readline(), b'data: first\n')
self.assertLess(time.monotonic() - started, .65)
self.assertIn(b'data: second', response.read())
connection.close()
class PortabilityTest(unittest.TestCase):
def test_port_probe_rejects_live_listener_and_allows_restart(self):
import control
with socket.socket() as listener:
listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
listener.bind(('127.0.0.1', 0))
port = listener.getsockname()[1]
listener.listen()
with self.assertRaises(RuntimeError):
control.free_port(port)
accepted, _ = listener.accept()
accepted.close()
control.free_port(port)
def test_process_identity_requires_start_time_executable_and_arguments(self):
import psutil
import control
process = psutil.Process()
info = control.record(process, process.cmdline())
self.assertTrue(control.matches(info))
self.assertFalse(control.matches({**info, 'created': info['created'] - 10}))
self.assertFalse(control.matches({**info, 'executable': '/unrelated/program'}))
self.assertFalse(control.matches({**info, 'args': ['unrelated-command']}))
def test_posix_permissions_are_not_relaxed(self):
if os.name != 'nt':
self.assertFalse(runtime_platform.private_bits(0o100600))
self.assertTrue(runtime_platform.private_bits(0o100640))
def test_lock_excludes_another_process(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / 'lock'
fd = os.open(path, os.O_CREAT | os.O_RDWR, 0o600)
runtime_platform.fcntl.flock(fd, runtime_platform.fcntl.LOCK_EX | runtime_platform.fcntl.LOCK_NB)
code = ('import sys,os;sys.path.insert(0,sys.argv[1]);import runtime_platform as h;'
'fd=os.open(sys.argv[2],os.O_RDWR);h.fcntl.flock(fd,h.fcntl.LOCK_EX|h.fcntl.LOCK_NB)')
result = subprocess.run([sys.executable, '-c', code, str(HERE), str(path)], capture_output=True)
self.assertNotEqual(result.returncode, 0)
os.close(fd)
def test_source_adapter_preserves_mac_source(self):
path = HERE.parents[1] / 'scripts/nao_quality_runtime.py'
source = path.read_text()
adapted = adapt_runtime(source)
self.assertIn('import fcntl', source)
self.assertIn('from runtime_platform import fcntl', adapted)
self.assertIn('_host.private_bits(directory_info.st_mode)', adapted)
self.assertIn('_host.killpg', adapted)
compile(adapted, str(path), 'exec')
self.assertEqual(path.read_text(), source)
if __name__ == '__main__':
unittest.main()
+175
View File
@@ -0,0 +1,175 @@
"""Offline tests for the signed Windows update publisher."""
import base64
import hashlib
import json
from pathlib import Path
import tempfile
import sys
import unittest
import zipfile
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
sys.path.insert(0, str(Path(__file__).resolve().parent))
import publish_update
import update_manager
class PublishUpdateTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.base = Path(self.temp.name)
self.root = self.base / 'windows-bundle'
self.root.mkdir()
self.output = self.base / 'published'
self.key_path = self.base / 'release-private-key.pem'
self.private = Ed25519PrivateKey.generate()
self.key_path.write_bytes(self.private.private_bytes(
serialization.Encoding.PEM,
serialization.PrivateFormat.PKCS8,
serialization.NoEncryption(),
))
public_key_path = self.root / update_manager.KEY
public_key_path.parent.mkdir(parents=True, exist_ok=True)
public_key_path.write_bytes(self.private.public_key().public_bytes(
serialization.Encoding.PEM,
serialization.PublicFormat.SubjectPublicKeyInfo,
))
self.data = {
'app/backend.jar': b'backend program',
'app/web/index.html': b'<html>program</html>',
'deploy/offline/control.py': b'control program',
'deploy/offline/update_manager.py': b'update manager program',
'deploy/offline/publish_update.py': b'publisher program',
'deploy/offline/windows-start.ps1': b'start script',
'deploy/offline/README.md': b'package instructions',
'backend/src/main/resources/analysis-runtime-policy.json': b'{"enabled":false}',
'scripts/runtime.py': b'helper program',
'ai/nao-quality/app.py': b'ai program',
'runtimes/python/python.exe': b'local runtime must not ship in an update',
'migration/manifest.json': b'private migration data',
'deploy/bootstrap/update-recovery.py': b'immutable bootstrap',
'.runtime/deployment.json': b'{"mysqlPassword":"private"}',
'runtimedata/archive/source.xlsx': b'user business data',
update_manager.KEY: public_key_path.read_bytes(),
}
for name, value in self.data.items():
target = self.root.joinpath(*name.split('/'))
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes(value)
self.manifest = {
'platform': 'windows',
'architecture': 'x86_64',
'updateVersion': '2026.11.04.1',
'updateSequence': 1,
'runtimes': [{'name': 'python', 'version': '3.11.15'}],
'pythonWheels': [{'file': 'cryptography.whl', 'sha256': 'a' * 64}],
'files': [
{'path': name, 'sha256': hashlib.sha256(value).hexdigest()}
for name, value in self.data.items()
],
}
(self.root / 'manifest.json').write_text(json.dumps(self.manifest), encoding='utf-8')
def build(self, **changes):
args = {
'root': self.root,
'version': '2026.11.04.2',
'sequence': 2,
'private_key_path': self.key_path,
'output_dir': self.output,
}
args.update(changes)
return publish_update.publish(**args)
def test_release_is_signed_program_snapshot_with_runtime_fingerprint(self):
artifacts = self.build()
self.assertTrue(artifacts.archive.is_file())
self.assertTrue(artifacts.latest.is_file())
envelope = update_manager.read(artifacts.latest)
release = update_manager.validate_release(envelope, self.root)
raw = base64.b64decode(envelope['payload'], validate=True)
self.private.public_key().verify(base64.b64decode(envelope['signature'], validate=True), raw)
self.assertEqual('2026.11.04.2', release['version'])
self.assertEqual(2, release['sequence'])
self.assertEqual(update_manager.runtime_fingerprint(self.manifest), release['runtimeFingerprint'])
self.assertNotIn('bundleManifest', release)
included = {entry['path'] for entry in release['files']}
self.assertEqual({name for name in self.data if update_manager.program_path(name)}, included)
self.assertFalse(included & {
'runtimes/python/python.exe',
'migration/manifest.json',
'deploy/bootstrap/update-recovery.py',
'.runtime/deployment.json',
'runtimedata/archive/source.xlsx',
update_manager.KEY,
})
with zipfile.ZipFile(artifacts.archive) as archive:
self.assertEqual(included, set(archive.namelist()))
self.assertEqual(self.data['app/backend.jar'], archive.read('app/backend.jar'))
self.assertEqual(hashlib.sha256(artifacts.archive.read_bytes()).hexdigest(), release['archive']['sha256'])
def test_rejects_wrong_key_stale_sequence_and_package_mutation(self):
with self.assertRaisesRegex(publish_update.PublishError, '不匹配'):
wrong_path = self.base / 'wrong-private-key.pem'
wrong_path.write_bytes(Ed25519PrivateKey.generate().private_bytes(
serialization.Encoding.PEM,
serialization.PrivateFormat.PKCS8,
serialization.NoEncryption(),
))
self.build(private_key_path=wrong_path)
with self.assertRaisesRegex(publish_update.PublishError, '大于部署包当前序号'):
self.build(sequence=1)
(self.root / 'app/backend.jar').write_bytes(b'changed after manifest')
with self.assertRaisesRegex(publish_update.PublishError, '摘要不一致'):
self.build()
self.assertFalse(self.output.exists())
def test_allows_only_explicit_exact_baseline_and_still_rejects_lower_sequence(self):
with self.assertRaisesRegex(publish_update.PublishError, '大于部署包当前序号'):
self.build(version='2026.11.04.1', sequence=1)
with self.assertRaisesRegex(publish_update.PublishError, '完全一致'):
self.build(version='2026.11.04.2', sequence=1, allow_baseline=True)
baseline = self.build(version='2026.11.04.1', sequence=1, allow_baseline=True)
release = update_manager.validate_release(update_manager.read(baseline.latest), self.root)
self.assertEqual('2026.11.04.1', release['version'])
self.assertEqual(1, release['sequence'])
self.manifest['updateVersion'] = '2026.11.04.1'
self.manifest['updateSequence'] = 2
(self.root / 'manifest.json').write_text(json.dumps(self.manifest), encoding='utf-8')
self.output = self.base / 'lower-sequence-output'
with self.assertRaisesRegex(publish_update.PublishError, '不能低于'):
self.build(version='2026.11.04.1', sequence=1, allow_baseline=True)
def test_rejects_sensitive_program_path_and_missing_program_inventory(self):
sensitive_name = 'ai/nao-quality/private/model.json'
sensitive_path = self.root.joinpath(*sensitive_name.split('/'))
sensitive_path.parent.mkdir(parents=True, exist_ok=True)
sensitive_path.write_bytes(b'private model config')
self.manifest['files'].append({'path': sensitive_name, 'sha256': hashlib.sha256(sensitive_path.read_bytes()).hexdigest()})
(self.root / 'manifest.json').write_text(json.dumps(self.manifest), encoding='utf-8')
with self.assertRaisesRegex(publish_update.PublishError, '受保护'):
self.build()
self.manifest['files'].pop()
self.manifest['files'] = [entry for entry in self.manifest['files'] if entry['path'] != 'scripts/runtime.py']
(self.root / 'manifest.json').write_text(json.dumps(self.manifest), encoding='utf-8')
with self.assertRaisesRegex(publish_update.PublishError, '完整程序文件清单'):
self.build()
def test_refuses_key_inside_source_or_output_and_refuses_overwrite(self):
with self.assertRaisesRegex(publish_update.PublishError, '部署包和更新产物目录之外'):
self.build(private_key_path=self.root / 'manifest.json')
with self.assertRaisesRegex(publish_update.PublishError, '部署包目录之外'):
self.build(output_dir=self.root / 'updates')
self.build()
original_latest = (self.output / 'latest.json').read_bytes()
with self.assertRaisesRegex(publish_update.PublishError, '同名发布文件'):
self.build()
self.assertEqual(original_latest, (self.output / 'latest.json').read_bytes())
if __name__ == '__main__':
unittest.main()
+339
View File
@@ -0,0 +1,339 @@
import hashlib
import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile
import textwrap
import time
import unittest
from unittest.mock import patch
import psutil
sys.path.insert(0, str(Path(__file__).resolve().parent))
import update_bootstrap as bootstrap
class ImmutableRecoveryTest(unittest.TestCase):
def fixture(self, folder):
root = Path(folder)
directory = root / '.runtime/update'
backup = directory / 'backup-20261104T120000Z-abcdef12'
(backup / 'program/deploy/offline').mkdir(parents=True)
(root / 'deploy/offline').mkdir(parents=True)
old = {
'app/backend.jar': b'verified old backend jar',
'deploy/offline/control.py': b'VALID_OLD_VERSION = True\n',
'deploy/offline/update_manager.py': b'def recover_locked(root):\n (root / ".runtime/restored").write_text("done")\n return {"recovered": True}\n',
}
entries = []
for name, content in old.items():
path = backup / 'program' / name
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(content)
(root / name).parent.mkdir(parents=True, exist_ok=True)
(root / name).write_bytes(b'def broken syntax\n')
entries.append({'path': name, 'sha256': hashlib.sha256(content).hexdigest()})
(backup / 'manifest.json').write_text(json.dumps({'files': entries}))
(backup / 'program.json').write_text(json.dumps({'all': list(old), 'present': list(old)}))
journal = {'phase': 'installing', 'databaseBackedUp': True, 'backup': backup.name,
'backupManifestSha256': bootstrap.digest(backup / 'manifest.json'),
'programInventorySha256': bootstrap.digest(backup / 'program.json')}
(directory / 'journal.json').write_text(json.dumps(journal))
return root, directory, backup, journal, old
def runtime_aliases(self, root):
for role in ('supervisor', 'backend'):
executable = bootstrap.runtime_executable(root, role)
executable.parent.mkdir(parents=True, exist_ok=True)
executable.symlink_to(psutil.Process().exe())
def record(self, process, name=None):
observed = psutil.Process(process.pid)
value = {'pid': process.pid, 'created': observed.create_time(),
'executable': str(Path(observed.exe()).resolve()), 'args': observed.cmdline()[1:]}
if name:
value['name'] = name
return value
def stop_test_child(self, process):
if process.poll() is not None:
return
descendants = []
try:
descendants = psutil.Process(process.pid).children(recursive=True)
except psutil.Error:
pass
process.terminate()
try:
process.wait(timeout=5)
except subprocess.TimeoutExpired:
process.kill()
process.wait(timeout=5)
for child in descendants:
try:
if child.is_running() and child.status() != psutil.STATUS_ZOMBIE:
child.kill()
except psutil.Error:
pass
def wait_file(self, path, process):
deadline = time.monotonic() + 10
while time.monotonic() < deadline:
if path.exists():
return
self.assertIsNone(process.poll(), 'Owned test child exited before readiness')
time.sleep(.02)
self.fail('Owned test child did not become ready')
def backend_script(self, root):
path = root / 'backend-test.py'
path.write_text(textwrap.dedent('''
import hashlib, os, signal, sys, time
from pathlib import Path
root = Path(__file__).parent
jar = root / 'app/backend.jar'
held = jar.open('rb')
def stop(*ignored):
(root / '.runtime/backend-stopped-before-copy').write_text(hashlib.sha256(jar.read_bytes()).hexdigest())
held.close()
raise SystemExit(0)
signal.signal(signal.SIGTERM, stop)
(root / '.runtime/backend-ready').write_text(str(os.getpid()))
while True:
time.sleep(.02)
'''))
return path
def start_backend(self, root):
self.runtime_aliases(root)
script = self.backend_script(root)
process = subprocess.Popen([str(bootstrap.runtime_executable(root, 'backend')), str(script), '-jar', str(root / 'app/backend.jar')],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
self.addCleanup(self.stop_test_child, process)
self.wait_file(root / '.runtime/backend-ready', process)
return process
def start_supervisor(self, root):
self.runtime_aliases(root)
self.backend_script(root)
script = root / 'deploy/offline/control.py'
script.write_text(textwrap.dedent('''
import json, os, signal, subprocess, sys, time
from pathlib import Path
import psutil
root = Path(__file__).resolve().parents[2]
token = 'a' * 48
java = root / 'runtimes/java/bin' / ('java.exe' if os.name == 'nt' else 'java')
child = subprocess.Popen([str(java), str(root / 'backend-test.py'), '-jar', str(root / 'app/backend.jar')])
def record(pid):
process = psutil.Process(pid)
return {'pid': pid, 'created': process.create_time(), 'executable': str(Path(process.exe()).resolve()), 'args': process.cmdline()[1:]}
state = {'supervisor': record(os.getpid()), 'token': token, 'children': [{**record(child.pid), 'name': 'backend'}], 'ready': True}
def stop(*ignored):
child.terminate()
child.wait(timeout=5)
(root / '.runtime/supervisor-stopped').write_text('done')
(root / '.runtime/service.json').unlink(missing_ok=True)
raise SystemExit(0)
signal.signal(signal.SIGTERM, stop)
(root / '.runtime/service.json').write_text(json.dumps(state))
(root / '.runtime/supervisor-ready').write_text('ready')
while True:
try:
request = json.loads((root / '.runtime/stop-request.json').read_text())
if request.get('token') == token:
(root / '.runtime/token-observed').write_text(token)
stop()
except (FileNotFoundError, ValueError):
pass
time.sleep(.02)
'''))
process = subprocess.Popen([str(bootstrap.runtime_executable(root, 'supervisor')), str(script), 'serve'],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
self.addCleanup(self.stop_test_child, process)
self.wait_file(root / '.runtime/supervisor-ready', process)
self.wait_file(root / '.runtime/backend-ready', process)
return process
def test_recovers_before_importing_broken_control_and_updater(self):
with tempfile.TemporaryDirectory() as folder:
root, directory, backup, journal, old = self.fixture(folder)
result = bootstrap.recover(root)
self.assertTrue(result['recovered'])
self.assertEqual((root / '.runtime/restored').read_text(), 'done')
for name, content in old.items():
self.assertEqual((root / name).read_bytes(), content)
self.assertFalse((directory / 'worker.json').exists())
def test_corrupt_backup_is_rejected_before_any_replacement(self):
with tempfile.TemporaryDirectory() as folder:
root, directory, backup, journal, old = self.fixture(folder)
(backup / 'program/deploy/offline/control.py').write_text('corruption')
with self.assertRaises(RuntimeError):
bootstrap.recover(root)
self.assertEqual((root / 'deploy/offline/control.py').read_text(), 'def broken syntax\n')
def test_active_worker_cannot_be_overridden(self):
with tempfile.TemporaryDirectory() as folder:
root, directory, backup, journal, old = self.fixture(folder)
(directory / 'worker.json').write_text(json.dumps({'pid': os.getpid(), 'created': psutil.Process().create_time()}))
with self.assertRaises(RuntimeError):
bootstrap.recover(root)
self.assertFalse((root / '.runtime/restored').exists())
def test_program_protection_and_no_update_are_read_only(self):
for name in ('deploy/bootstrap/windows-start.ps1', 'deploy/offline/update-public-key.pem', '.runtime/deployment.json', 'runtimedata/mysql/file'):
self.assertFalse(bootstrap.program(name))
with tempfile.TemporaryDirectory() as folder:
root = Path(folder)
self.assertEqual(bootstrap.recover(root), {'recovered': False})
self.assertFalse((root / '.runtime').exists())
def test_orphaned_backend_stops_before_jar_copy_after_worker_crash(self):
with tempfile.TemporaryDirectory() as folder:
root, directory, _, _, old = self.fixture(folder)
child = self.start_backend(root)
state = {'supervisor': {'pid': 999999999, 'created': 1, 'executable': str(sys.executable),
'args': [str(root / 'deploy/offline/control.py'), 'serve']}, 'token': 'b' * 48,
'children': [self.record(child, 'backend')]}
(root / '.runtime/service.json').write_text(json.dumps(state))
(directory / 'worker.json').write_text(json.dumps({'pid': 999999998, 'created': 1}))
before = bootstrap.digest(root / 'app/backend.jar')
real_copy = bootstrap.atomic_copy
def checked_copy(source, target):
self.assertIsNotNone(child.poll(), 'A mutable program was copied while the old backend still held its JAR')
return real_copy(source, target)
with patch.object(bootstrap, 'atomic_copy', side_effect=checked_copy):
self.assertTrue(bootstrap.recover(root)['recovered'])
self.assertEqual(before, (root / '.runtime/backend-stopped-before-copy').read_text())
self.assertEqual(old['app/backend.jar'], (root / 'app/backend.jar').read_bytes())
def test_live_supervisor_receives_matching_stop_token_and_children_exit_before_copy(self):
with tempfile.TemporaryDirectory() as folder:
root, directory, _, _, _ = self.fixture(folder)
supervisor = self.start_supervisor(root)
(root / 'deploy/offline/control.py').write_text('def broken syntax\n')
before = bootstrap.digest(root / 'app/backend.jar')
result = bootstrap.recover(root)
self.assertTrue(result['recovered'])
self.assertIsNotNone(supervisor.poll())
self.assertEqual('a' * 48, (root / '.runtime/token-observed').read_text())
self.assertEqual(before, (root / '.runtime/backend-stopped-before-copy').read_text())
self.assertFalse((root / '.runtime/stop-request.json').exists())
def test_stale_record_pointing_at_an_unrelated_live_pid_never_kills_or_replaces(self):
with tempfile.TemporaryDirectory() as folder:
root, _, _, _, _ = self.fixture(folder)
self.runtime_aliases(root)
child = subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(60)'], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
self.addCleanup(self.stop_test_child, child)
info = self.record(child, 'backend')
info['created'] -= 60
(root / '.runtime/service.json').write_text(json.dumps({'children': [info]}))
with self.assertRaisesRegex(RuntimeError, '归属'):
bootstrap.recover(root)
self.assertIsNone(child.poll())
self.assertEqual('def broken syntax\n', (root / 'deploy/offline/control.py').read_text())
self.assertFalse((root / '.runtime/restored').exists())
def test_unregistered_root_backend_is_refused_without_signalling(self):
with tempfile.TemporaryDirectory() as folder:
root, _, _, _, _ = self.fixture(folder)
child = self.start_backend(root)
with self.assertRaisesRegex(RuntimeError, '未登记'):
bootstrap.recover(root)
self.assertIsNone(child.poll())
self.assertEqual('def broken syntax\n', (root / 'deploy/offline/control.py').read_text())
def test_other_deployment_root_cannot_be_claimed_by_a_valid_live_pid_record(self):
with tempfile.TemporaryDirectory() as first, tempfile.TemporaryDirectory() as second:
root, _, _, _, _ = self.fixture(first)
other, _, _, _, _ = self.fixture(second)
self.runtime_aliases(root)
child = self.start_backend(other)
(root / '.runtime/service.json').write_text(json.dumps({'children': [self.record(child, 'backend')]}))
with self.assertRaisesRegex(RuntimeError, '归属'):
bootstrap.recover(root)
self.assertIsNone(child.poll())
self.assertFalse((root / '.runtime/restored').exists())
def test_invalid_supervisor_token_does_not_signal_any_owned_process(self):
with tempfile.TemporaryDirectory() as folder:
root, _, _, _, _ = self.fixture(folder)
supervisor = self.start_supervisor(root)
state = json.loads((root / '.runtime/service.json').read_text())
state['token'] = 'invalid-token'
(root / '.runtime/service.json').write_text(json.dumps(state))
with self.assertRaisesRegex(RuntimeError, '令牌'):
bootstrap.recover(root)
self.assertIsNone(supervisor.poll())
self.assertFalse((root / '.runtime/stop-request.json').exists())
def test_dead_pid_records_do_not_block_safe_recovery(self):
with tempfile.TemporaryDirectory() as folder:
root, _, _, _, _ = self.fixture(folder)
state = {'children': [{'pid': 999999999, 'created': 1, 'executable': str(sys.executable),
'args': ['missing'], 'name': 'backend'}]}
(root / '.runtime/service.json').write_text(json.dumps(state))
self.assertTrue(bootstrap.recover(root)['recovered'])
def test_a_process_owned_by_another_user_is_never_stopped(self):
with tempfile.TemporaryDirectory() as folder:
root, _, _, _, _ = self.fixture(folder)
child = self.start_backend(root)
(root / '.runtime/service.json').write_text(json.dumps({'children': [self.record(child, 'backend')]}))
with patch.object(bootstrap, 'current_user', return_value=False), self.assertRaisesRegex(RuntimeError, '归属'):
bootstrap.recover(root)
self.assertIsNone(child.poll())
self.assertFalse((root / '.runtime/restored').exists())
@unittest.skipIf(os.name == 'nt', 'Host-side immutable-client fixture uses a POSIX shebang; real Windows binaries need native acceptance')
def test_orphaned_mysql_uses_immutable_shutdown_client_and_is_not_force_killed(self):
with tempfile.TemporaryDirectory() as folder:
root, _, _, _, _ = self.fixture(folder)
executable = bootstrap.runtime_executable(root, 'mysql')
executable.parent.mkdir(parents=True)
executable.symlink_to(psutil.Process().exe())
server = root / 'mysql-test.py'
server.write_text(textwrap.dedent('''
import os, signal, time
from pathlib import Path
root = Path(__file__).parent
def stop(*ignored):
(root / '.runtime/mysql-stopped-cleanly').write_text('done')
raise SystemExit(0)
signal.signal(signal.SIGTERM, stop)
(root / '.runtime/mysql-test-pid').write_text(str(os.getpid()))
while True:
time.sleep(.02)
'''))
child = subprocess.Popen([str(executable), str(server), '--datadir=' + str(root / 'runtimedata/mysql'), '--port=54489'],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
self.addCleanup(self.stop_test_child, child)
self.wait_file(root / '.runtime/mysql-test-pid', child)
admin = executable.parent / 'mysqladmin'
admin.write_text('#!' + psutil.Process().exe() + '\n' + textwrap.dedent('''
import os, signal, sys
from pathlib import Path
root = Path(__file__).resolve().parents[3]
assert os.environ['MYSQL_PWD'] == 'isolated_password_value_1234'
assert '--no-defaults' in sys.argv and '--port=54489' in sys.argv and sys.argv[-1] == 'shutdown'
assert not any('password' in arg.lower() for arg in sys.argv)
(root / '.runtime/immutable-mysqladmin-called').write_text('done')
os.kill(int((root / '.runtime/mysql-test-pid').read_text()), signal.SIGTERM)
'''))
admin.chmod(0o700)
(root / '.runtime/deployment.json').write_text(json.dumps({'mysqlPort': 54489, 'rootPassword': 'isolated_password_value_1234'}))
(root / '.runtime/service.json').write_text(json.dumps({'children': [self.record(child, 'mysql')]}))
with patch.object(psutil.Process, 'kill', side_effect=AssertionError('MySQL must not be force killed')):
self.assertTrue(bootstrap.recover(root)['recovered'])
self.assertIsNotNone(child.poll())
self.assertTrue((root / '.runtime/immutable-mysqladmin-called').exists())
self.assertTrue((root / '.runtime/mysql-stopped-cleanly').exists())
if __name__ == '__main__':
unittest.main()
+303
View File
@@ -0,0 +1,303 @@
"""Offline security and rollback tests. No network, production DB or services."""
import base64
import copy
import hashlib
import importlib.util
import json
from pathlib import Path
import shutil
import stat
import sys
import tempfile
import types
import unittest
from unittest.mock import patch
import zipfile
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
from cryptography.hazmat.primitives import serialization
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE))
import update_manager as update
class UpdateTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name) / 'bundle'
self.root.mkdir()
self.private = Ed25519PrivateKey.generate()
self.files = {
'app/backend.jar': b'old backend',
'app/web/index.html': b'old web',
'app/web/obsolete.js': b'old asset',
'deploy/offline/control.py': b'old supervisor',
'deploy/offline/update_manager.py': b'old updater',
'backend/src/main/resources/analysis-runtime-policy.json': b'{}',
update.KEY: self.private.public_key().public_bytes(serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo),
'runtimes/python/python.exe': b'unchanged runtime',
'migration/snapshot.sql': b'private migration data',
}
for name, data in self.files.items():
self.write(name, data)
self.manifest = {
'platform': 'windows', 'architecture': 'x86_64',
'runtimes': [{'name': 'python', 'version': '3.11.15'}],
'pythonWheels': [{'file': 'cryptography.whl', 'sha256': 'a' * 64}],
'files': [{'path': k, 'sha256': hashlib.sha256(v).hexdigest()} for k, v in self.files.items()],
'updateSequence': 5, 'updateVersion': '2026.11.03',
}
update.atomic(self.root / 'manifest.json', self.manifest)
self.config = {'mysqlDatabase': 'quality_restore_' + 'a' * 24, 'archiveDirectory': 'runtimedata/archive/restore-old',
'mysqlPort': 53306, 'rootPassword': 'privateRootPasswordValue', 'mysqlPassword': 'privateAppPasswordValue'}
update.atomic(self.root / '.runtime/deployment.json', self.config)
self.write('.runtime/analysis-api-settings.json', b'{"apiKey":"keep-private"}')
self.program = {k: b'new ' + v for k, v in self.files.items() if update.program_path(k) and k != 'app/web/obsolete.js'}
self.release = {'schema': 1, 'version': '2026.11.04', 'sequence': 6, 'platform': 'windows-x64',
'runtimeFingerprint': update.runtime_fingerprint(self.manifest),
'archive': {'file': 'quality-update-2026.11.04.zip', 'size': 123, 'sha256': 'a' * 64},
'files': [{'path': k, 'size': len(v), 'sha256': hashlib.sha256(v).hexdigest()} for k, v in self.program.items()]}
def write(self, name, data):
path = self.root / name
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(data)
def envelope(self, release=None, private=None):
raw = update.canonical(release or self.release)
return {'payload': base64.b64encode(raw).decode(), 'signature': base64.b64encode((private or self.private).sign(raw)).decode()}
def validate(self, release=None):
return update.validate_release(self.envelope(release), self.root)
def archive(self, entries=None):
archive = Path(self.temp.name) / 'update.zip'
with zipfile.ZipFile(archive, 'w') as out:
for name, data in (entries or self.program).items():
out.writestr(name, data)
return archive
def test_valid_signature_and_runtime_identity(self):
self.assertEqual('2026.11.04', self.validate()['version'])
self.assertEqual(update.runtime_fingerprint(self.manifest), update.runtime_fingerprint(copy.deepcopy(self.manifest)))
def test_wrong_signature_key_and_modified_payload_are_rejected(self):
with self.assertRaises(update.UpdateError):
update.validate_release(self.envelope(private=Ed25519PrivateKey.generate()), self.root)
envelope = self.envelope()
envelope['payload'] = base64.b64encode(update.canonical({**self.release, 'sequence': 999})).decode()
with self.assertRaises(update.UpdateError):
update.validate_release(envelope, self.root)
def test_immutable_trust_anchor_checked_against_local_manifest(self):
self.write(update.KEY, Ed25519PrivateKey.generate().public_key().public_bytes(serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo))
with self.assertRaises(update.UpdateError):
self.validate()
def test_runtime_mismatch_requires_full_bundle(self):
release = {**self.release, 'runtimeFingerprint': 'b' * 64}
with self.assertRaisesRegex(update.UpdateError, '完整安装包'):
self.validate(release)
def test_forbidden_destinations_and_trust_key_are_never_update_files(self):
for name in ['.runtime/deployment.json', 'runtimedata/mysql/table.ibd', 'migration/snapshot.sql',
'runtimes/python/python.exe', update.KEY, 'manifest.json', 'backend/src/private.key']:
with self.subTest(name=name):
release = copy.deepcopy(self.release)
release['files'].append({'path': name, 'size': 0, 'sha256': 'a' * 64})
with self.assertRaises(update.UpdateError):
self.validate(release)
def test_windows_aliases_traversal_and_case_collisions_rejected(self):
for name in ['../evil', 'app/web/../evil', 'app/web/C:evil', 'app\\web\\evil', '/absolute',
'app/web/NUL.txt', 'app/web/trailing.', 'app/web/double//slash', 'app/web/control\x01']:
with self.subTest(name=name), self.assertRaises(update.UpdateError):
update.safe_path(name)
release = copy.deepcopy(self.release)
release['files'].append({**release['files'][0], 'path': release['files'][0]['path'].upper()})
with self.assertRaises(update.UpdateError):
self.validate(release)
def test_destination_symlink_rejected(self):
target = self.root / 'app/web/link'
target.symlink_to(Path(self.temp.name), target_is_directory=True)
with self.assertRaises(update.UpdateError):
update.destination(self.root, 'app/web/link/escape.html')
def test_missing_required_program_or_deployment_manifest_rejected(self):
release = copy.deepcopy(self.release)
release['files'] = [e for e in release['files'] if e['path'] != 'backend/src/main/resources/analysis-runtime-policy.json']
with self.assertRaises(update.UpdateError):
self.validate(release)
with self.assertRaisesRegex(update.UpdateError, '部署数据'):
self.validate({**self.release, 'bundleManifest': self.manifest})
def test_zip_inventory_and_hash_enforced(self):
stage = Path(self.temp.name) / 'good'
update.unpack(self.archive(), stage, self.release)
self.assertEqual(self.program['app/backend.jar'], (stage / 'app/backend.jar').read_bytes())
bad = dict(self.program)
bad['../evil'] = b'escape'
with self.assertRaises(update.UpdateError):
update.unpack(self.archive(bad), Path(self.temp.name) / 'escape', self.release)
bad = dict(self.program)
bad['app/backend.jar'] = b'x' * len(bad['app/backend.jar'])
with self.assertRaisesRegex(update.UpdateError, '校验失败'):
update.unpack(self.archive(bad), Path(self.temp.name) / 'hash', self.release)
def test_zip_symlink_rejected(self):
archive = Path(self.temp.name) / 'symlink.zip'
info = zipfile.ZipInfo('app/backend.jar')
info.external_attr = (stat.S_IFLNK | 0o777) << 16
with zipfile.ZipFile(archive, 'w') as out:
out.writestr(info, self.program['app/backend.jar'])
with self.assertRaises(update.UpdateError):
update.unpack(archive, Path(self.temp.name) / 'links', self.release)
def test_snapshot_replace_preserves_local_data_and_manifest_entries(self):
backup = self.root / '.runtime/update/backup-test'
backup.mkdir(parents=True)
stage = Path(self.temp.name) / 'stage'
update.unpack(self.archive(), stage, self.release)
update.backup_program(self.root, backup, self.release)
update.replace_program(self.root, stage, backup, self.release)
result = update.read(self.root / 'manifest.json')
self.assertEqual(6, result['updateSequence'])
self.assertEqual(self.config, update.read(self.root / '.runtime/deployment.json'))
self.assertFalse((self.root / 'app/web/obsolete.js').exists())
for name in (update.KEY, 'runtimes/python/python.exe', 'migration/snapshot.sql'):
self.assertEqual(self.files[name], (self.root / name).read_bytes())
self.assertIn(next(e for e in self.manifest['files'] if e['path'] == name), result['files'])
update.restore_program(self.root, backup)
self.assertEqual(self.manifest, update.read(self.root / 'manifest.json'))
self.assertEqual(self.files['app/web/obsolete.js'], (self.root / 'app/web/obsolete.js').read_bytes())
def worker_harness(self, fail_verify=False):
directory = self.root / '.runtime/update'
update.atomic(directory / 'request.json', {'token': 'a' * 32})
archive = self.archive()
envelope = self.envelope()
original_download = update.download
calls, restores, queries = [], [], []
def fake_download(url, target=None, *args):
if target:
shutil.copyfile(archive, target)
return b''
return json.dumps(envelope).encode()
def fake_control(root, action, database_only=False, timeout=0):
calls.append((action, database_only))
if action == 'verify' and fail_verify:
raise update.UpdateError('模拟新版启动校验失败。')
return types.SimpleNamespace(pid=1234)
def fake_mysql(root, config, binary='mysql', database=None, stdin=None, stdout=None):
if binary == 'mysqldump':
stdout.write(b'-- database snapshot\n' + b'SELECT 1;\n' * 30)
else:
restores.append((database, stdin.read()))
process = types.SimpleNamespace(parents=lambda: [], create_time=lambda: 1,
is_running=lambda: True, status=lambda: 'running')
with patch.object(update.psutil, 'pid_exists', return_value=False), \
patch.object(update.psutil, 'Process', return_value=process), \
patch.object(update, 'download', side_effect=fake_download), \
patch.object(update, 'controller', side_effect=fake_control), \
patch.object(update, 'wait_ready'), \
patch.object(update, 'mysql', side_effect=fake_mysql), \
patch.object(update, 'sql', side_effect=lambda root, config, query: queries.append(query)):
update.worker(987654, 'a' * 32, self.root)
return calls, restores, queries
def test_complete_update_orchestration_with_isolated_database_backup(self):
calls, restores, queries = self.worker_harness()
self.assertEqual([('stop', False), ('serve', True), ('stop', False), ('verify', False), ('serve', False)], calls)
self.assertFalse(restores)
self.assertEqual('completed', update.read(self.root / '.runtime/update/status.json')['state'])
self.assertEqual('complete', update.read(self.root / '.runtime/update/journal.json')['phase'])
self.assertEqual(6, update.read(self.root / 'manifest.json')['updateSequence'])
def test_failed_update_restores_old_program_and_new_rollback_database(self):
self.config['restoredFromMigration'] = 'isolated-test-migration'
update.atomic(self.root / '.runtime/deployment.json', self.config)
update.atomic(self.root / '.runtime/migration-receipt.json', {'migrationId': 'isolated-test-migration',
'database': self.config['mysqlDatabase'], 'migrationManifestSha256': 'b' * 64})
calls, restores, queries = self.worker_harness(fail_verify=True)
self.assertEqual(self.manifest, update.read(self.root / 'manifest.json'))
config = update.read(self.root / '.runtime/deployment.json')
self.assertRegex(config['mysqlDatabase'], r'^quality_restore_[a-f0-9]{24}$')
self.assertNotEqual(self.config['mysqlDatabase'], config['mysqlDatabase'])
self.assertEqual(self.config['archiveDirectory'], config['archiveDirectory'])
self.assertEqual(config['mysqlDatabase'], restores[0][0])
self.assertEqual(config['mysqlDatabase'], update.read(self.root / '.runtime/migration-receipt.json')['database'])
self.assertTrue(any('CREATE DATABASE' in q for q in queries))
self.assertFalse(any('DROP' in q for q in queries))
self.assertEqual('failed', update.read(self.root / '.runtime/update/status.json')['state'])
self.assertIn('已恢复原版本', update.read(self.root / '.runtime/update/status.json')['message'])
self.assertEqual('rolledBack', update.read(self.root / '.runtime/update/journal.json')['phase'])
def test_recovery_blocks_live_worker_and_rejects_untrusted_backup_path(self):
directory = self.root / '.runtime/update'
update.atomic(directory / 'journal.json', {'phase': 'installing', 'backup': '../../escape'})
with patch.object(update, 'live', return_value=False), self.assertRaises(update.UpdateError):
update.recover(self.root)
with patch.object(update, 'live', return_value=True), self.assertRaisesRegex(update.UpdateError, '正在进行'):
update.recover(self.root)
def test_arbitrary_download_url_is_rejected_before_network(self):
for url in ['http://quality.lyunlab.me/quality-updates/windows-x64/latest.json', 'https://evil.example/update.zip',
update.BASE + '../private.zip', update.BASE + 'quality-update-x.zip?token=x']:
with self.subTest(url=url), self.assertRaises(update.UpdateError):
update.download(url)
def test_source_is_disabled_and_status_never_exposes_private_fields(self):
update.atomic(self.root / '.runtime/update/status.json', {'state': 'idle', 'rootPassword': 'secret', 'backupPath': '/private'})
with patch.object(update.os, 'name', 'posix'):
self.assertFalse(update.enabled(self.root))
result = update.status(self.root)
self.assertNotIn('rootPassword', result)
self.assertNotIn('backupPath', result)
def test_recovery_and_worker_share_lock_and_live_identity(self):
with update.exclusive_update(self.root):
self.assertTrue(update.live(update.read(self.root / '.runtime/update/worker.json')))
with self.assertRaises(update.BusyUpdate):
update.recover(self.root)
# The lock remains authoritative if the identity file disappears.
(self.root / '.runtime/update/worker.json').unlink()
with self.assertRaises(update.BusyUpdate):
with update.exclusive_update(self.root):
self.fail('Concurrent update lock was acquired')
def test_atomic_copy_failure_preserves_existing_control_and_configuration(self):
source = Path(self.temp.name) / 'new-content'
source.write_bytes(b'new file content')
for name in ('deploy/offline/control.py', '.runtime/deployment.json', 'manifest.json'):
target = self.root / name
previous = target.read_bytes()
with self.subTest(name=name), patch.object(update.os, 'replace', side_effect=OSError('simulated interruption')):
with self.assertRaises(OSError):
update.copy_atomic(source, target)
self.assertEqual(previous, target.read_bytes())
self.assertFalse(list(target.parent.glob(target.name + '.update-*')))
def test_mysqldump_does_not_receive_mysql_only_connect_timeout_option(self):
with patch.object(update.subprocess, 'run', return_value=types.SimpleNamespace(returncode=0)) as run:
update.mysql(self.root, self.config, 'mysqldump', stdout=update.subprocess.DEVNULL)
self.assertFalse(any(arg.startswith('--connect-timeout') for arg in run.call_args.args[0]))
update.mysql(self.root, self.config, 'mysql')
self.assertIn('--connect-timeout=10', run.call_args.args[0])
def test_actual_dump_is_followed_by_a_second_recovery_space_check(self):
with patch.object(update.shutil, 'disk_usage', side_effect=[types.SimpleNamespace(free=10**15), types.SimpleNamespace(free=0)]):
calls, _, _ = self.worker_harness()
self.assertNotIn(('verify', False), calls)
self.assertEqual(self.manifest, update.read(self.root / 'manifest.json'))
self.assertIn('剩余磁盘空间不足', update.read(self.root / '.runtime/update/status.json')['message'])
if __name__ == '__main__':
unittest.main()
+3
View File
@@ -0,0 +1,3 @@
-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEAMPNTCeYpWygXdSxtaJzijXu0mKZF64lP7NcUuYZ4NcU=
-----END PUBLIC KEY-----
+357
View File
@@ -0,0 +1,357 @@
"""Immutable stage-zero recovery. Installed under deploy/bootstrap, never updated."""
from __future__ import annotations
import hashlib
import importlib.util
import json
import os
from pathlib import Path
import re
import secrets
import shutil
import subprocess
import sys
import time
import psutil # Bundled immutable Python environment, not an application module.
ROOT = Path(__file__).resolve().parents[2]
def read(path, default=None):
return json.loads(path.read_text(encoding='utf-8')) if path.exists() else default
def digest(path):
with path.open('rb') as stream:
return hashlib.file_digest(stream, 'sha256').hexdigest()
def atomic_copy(source, target):
target.parent.mkdir(parents=True, exist_ok=True)
temporary = target.with_name(target.name + '.recover-' + secrets.token_hex(6))
try:
with source.open('rb') as reader, temporary.open('xb') as writer:
shutil.copyfileobj(reader, writer)
writer.flush()
os.fsync(writer.fileno())
temporary.replace(target)
finally:
temporary.unlink(missing_ok=True)
def safe(root, name):
if (not isinstance(name, str) or not name or '\\' in name or ':' in name or name.startswith('/')
or any(part in ('', '.', '..') or part.endswith((' ', '.')) for part in name.split('/'))):
raise RuntimeError('更新恢复路径无效。')
target = root.joinpath(*name.split('/'))
for path in (target, *target.parents):
if path == root:
break
if path.is_symlink() or (path.exists() and getattr(path.lstat(), 'st_file_attributes', 0) & 0x400):
raise RuntimeError('更新恢复目录包含无效链接。')
if not target.resolve().is_relative_to(root.resolve()):
raise RuntimeError('更新恢复路径越界。')
return target
def program(name):
if name == 'deploy/offline/update-public-key.pem' or name.startswith('deploy/bootstrap/'):
return False
path = Path(name)
return (name == 'app/backend.jar' or name.startswith('app/web/')
or name == 'backend/src/main/resources/analysis-runtime-policy.json'
or name.startswith('scripts/') and path.suffix == '.py'
or name.startswith('ai/nao-quality/')
or name.startswith('deploy/offline/') and path.suffix in ('.py', '.ps1', '.md'))
def atomic_json(path, value):
temporary = path.with_name(path.name + '.stop-' + secrets.token_hex(6))
try:
with temporary.open('x', encoding='utf-8') as stream:
json.dump(value, stream)
stream.flush()
os.fsync(stream.fileno())
temporary.replace(path)
finally:
temporary.unlink(missing_ok=True)
def same_path(left, right):
return os.path.normcase(str(Path(left).resolve())) == os.path.normcase(str(Path(right).resolve()))
def runtime_executable(root, role):
suffix = '.exe' if os.name == 'nt' else ''
relative = {'supervisor': 'runtimes/python/' + ('python.exe' if os.name == 'nt' else 'bin/python3'),
'backend': 'runtimes/java/bin/java' + suffix, 'mysql': 'runtimes/mysql/bin/mysqld' + suffix}[role]
return root / relative
def bound_to_root(root, role, args):
if not isinstance(args, list) or not all(isinstance(arg, str) for arg in args):
return False
if role == 'supervisor':
return (len(args) in (2, 3) and same_path(args[0], root / 'deploy/offline/control.py')
and args[1] == 'serve' and (len(args) == 2 or args[2] == '--database-only'))
if role == 'backend':
return len(args) >= 2 and args[-2] == '-jar' and same_path(args[-1], root / 'app/backend.jar')
datadirs = [arg.split('=', 1)[1] for arg in args if arg.startswith('--datadir=')]
return len(datadirs) == 1 and same_path(datadirs[0], root / 'runtimedata/mysql')
def current_user(process):
if os.name == 'nt':
return process.username().casefold() == psutil.Process().username().casefold()
return process.uids().real == os.getuid()
def owned_process(root, role, info):
"""Return a fully verified owned PID, None for a dead PID, never guess."""
if (not isinstance(info, dict) or type(info.get('pid')) is not int or info['pid'] <= 0
or not isinstance(info.get('created'), (int, float))
or not isinstance(info.get('executable'), str) or not isinstance(info.get('args'), list)):
raise RuntimeError('本包服务进程记录无效,未停止任何不明进程。')
try:
process = psutil.Process(info['pid'])
if not process.is_running() or process.status() == psutil.STATUS_ZOMBIE:
return None
forbidden = {os.getpid(), *(parent.pid for parent in psutil.Process().parents())}
if (process.pid in forbidden or abs(process.create_time() - info['created']) >= .01
or not current_user(process) or not same_path(process.exe(), info['executable'])
or process.cmdline()[1:] != info['args']
or not same_path(info['executable'], runtime_executable(root, role))
or not bound_to_root(root, role, info['args'])):
raise RuntimeError('无法确认本包服务进程归属,未停止任何不明进程。')
return process
except psutil.NoSuchProcess:
return None
except psutil.Error:
raise RuntimeError('当前用户无法核验本包服务进程,恢复已安全停止。') from None
def root_markers(root, args):
"""Detect unregistered package processes without inspecting unrelated data."""
for arg in args:
if arg.startswith('--datadir='):
if same_path(arg.split('=', 1)[1], root / 'runtimedata/mysql'):
return True
elif Path(arg).is_absolute() and (same_path(arg, root / 'app/backend.jar')
or same_path(arg, root / 'deploy/offline/control.py')):
return True
return False
def refuse_unregistered(root, expected):
for process in psutil.process_iter():
try:
if process.pid == os.getpid() or process.status() == psutil.STATUS_ZOMBIE:
continue
args = process.cmdline()[1:]
if not root_markers(root, args):
continue
if process.pid not in expected:
raise RuntimeError('发现未登记的本包服务进程,已保留现场,请先检查该进程。')
except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess):
continue
def wait_owned(root, role, info, seconds):
deadline = time.monotonic() + seconds
while time.monotonic() < deadline:
if owned_process(root, role, info) is None:
return True
time.sleep(.05)
return owned_process(root, role, info) is None
def terminate_owned(root, role, info, seconds):
process = owned_process(root, role, info)
if process is None:
return
try:
process.terminate()
if wait_owned(root, role, info, seconds):
return
process = owned_process(root, role, info)
if process is not None:
process.kill()
if not wait_owned(root, role, info, 10):
raise RuntimeError('本包服务尚未退出,未替换任何程序文件。')
except psutil.NoSuchProcess:
return
except psutil.Error:
raise RuntimeError('当前用户无法停止本包服务,未替换任何程序文件。') from None
def shutdown_owned_mysql(root, info):
process = owned_process(root, 'mysql', info)
if process is None:
return
value = read(safe(root, '.runtime/deployment.json'), {})
port = value.get('mysqlPort')
if (type(port) is not int or not 1024 <= port <= 65535
or not re.fullmatch(r'[A-Za-z0-9_-]{20,100}', str(value.get('rootPassword', '')))
or info['args'].count('--port=' + str(port)) != 1):
raise RuntimeError('无法核验本包数据库关闭配置,未替换任何程序文件。')
executable = root / 'runtimes/mysql/bin' / ('mysqladmin.exe' if os.name == 'nt' else 'mysqladmin')
env = os.environ.copy()
env['MYSQL_PWD'] = value['rootPassword']
# Only the immutable bundled MySQL client is executed; no application code
# is loaded, and the current user's privileges are never elevated.
command = [str(executable), '--no-defaults', '--protocol=TCP', '--host=127.0.0.1',
'--port=' + str(port), '--user=root', '--connect-timeout=3', 'shutdown']
try:
subprocess.run(command, env=env, cwd=root, stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=20)
except (OSError, subprocess.SubprocessError):
raise RuntimeError('本包数据库无法安全关闭,未替换任何程序文件。') from None
if not wait_owned(root, 'mysql', info, 120):
raise RuntimeError('本包数据库仍在关闭,未替换任何程序文件,请稍后重新启动。')
def stop_owned_services(root, grace_seconds=200):
"""Immutable implementation of service.json + stop-request token contract.
A crashed worker can leave a healthy new supervisor/Java behind. Validate
every recorded identity before requesting shutdown; a live stale/reused PID
is an error rather than permission to terminate it. Java must be gone before
a JAR is replaced. An orphaned MySQL is shut down through its immutable
client, never forcibly killed on Windows.
"""
state = read(safe(root, '.runtime/service.json'), {})
if not isinstance(state, dict) or not isinstance(state.get('children', []), list):
raise RuntimeError('本包服务状态无效,未替换任何程序文件。')
entries = []
supervisor = state.get('supervisor')
if supervisor:
entries.append(('supervisor', supervisor))
for info in state.get('children', []):
if not isinstance(info, dict) or info.get('name') not in ('backend', 'mysql'):
raise RuntimeError('本包服务子进程记录无效,未替换任何程序文件。')
entries.append((info['name'], info))
if len(entries) > 20 or len({info.get('pid') for _, info in entries}) != len(entries):
raise RuntimeError('本包服务进程记录重复,未替换任何程序文件。')
# Validate all owners before the first stop request or signal is sent.
active = {(role, info['pid']): owned_process(root, role, info) for role, info in entries}
refuse_unregistered(root, {pid for (_, pid), process in active.items() if process is not None})
if supervisor and active.get(('supervisor', supervisor['pid'])) is not None:
token = state.get('token')
if not isinstance(token, str) or not re.fullmatch(r'[a-f0-9]{48}', token):
raise RuntimeError('本包服务停止令牌无效,未停止任何进程。')
atomic_json(safe(root, '.runtime/stop-request.json'), {'token': token})
if not wait_owned(root, 'supervisor', supervisor, grace_seconds):
terminate_owned(root, 'supervisor', supervisor, 10)
# A dead supervisor cannot finish its own finally block. Only independently
# verified package children may be stopped; unrelated/reused PIDs are refused.
for role, info in entries:
if role == 'backend':
terminate_owned(root, role, info, 35)
for role, info in entries:
if role == 'mysql':
shutdown_owned_mysql(root, info)
for role, info in entries:
if owned_process(root, role, info) is not None:
raise RuntimeError('本包服务尚未完全退出,未替换任何程序文件。')
refuse_unregistered(root, set())
stop = safe(root, '.runtime/stop-request.json')
if isinstance(state.get('token'), str) and read(stop, {}).get('token') == state['token']:
stop.unlink(missing_ok=True)
def restore_program_before_import(root, directory, journal):
name = journal.get('backup', '')
if not re.fullmatch(r'backup-[0-9]{8}T[0-9]{6}Z-[a-f0-9]{8}', name):
raise RuntimeError('更新恢复记录无效。')
backup = safe(directory, name)
manifest_file, inventory_file = backup / 'manifest.json', backup / 'program.json'
if (digest(manifest_file) != journal.get('backupManifestSha256')
or digest(inventory_file) != journal.get('programInventorySha256')):
raise RuntimeError('本机更新备份校验失败,请保留目录并联系管理员。')
manifest = read(manifest_file)
old = {entry['path']: entry for entry in manifest['files'] if program(entry['path'])}
inventory = read(inventory_file)
all_names, present = inventory['all'], set(inventory['present'])
if (not isinstance(all_names, list) or len(all_names) > 50000
or len(set(all_names)) != len(all_names) or not present.issubset(all_names)):
raise RuntimeError('更新恢复清单无效。')
for name in all_names:
if not program(name):
raise RuntimeError('更新恢复清单包含受保护路径。')
target = safe(root, name)
if name in present:
source = safe(backup / 'program', name)
if name not in old or not source.is_file() or digest(source) != old[name]['sha256']:
raise RuntimeError('原程序备份缺失或损坏。')
elif name in old:
raise RuntimeError('原程序备份不完整。')
# All backups are validated before any application file is replaced.
stop_owned_services(root)
for name in all_names:
target = safe(root, name)
if name in present:
atomic_copy(safe(backup / 'program', name), target)
else:
target.unlink(missing_ok=True)
atomic_copy(manifest_file, root / 'manifest.json')
def live(info):
try:
process = psutil.Process(info['pid'])
return (process.is_running() and process.status() != psutil.STATUS_ZOMBIE
and abs(process.create_time() - info['created']) < .01)
except (psutil.Error, ValueError, TypeError, KeyError):
return False
def recover(root=ROOT):
directory = safe(root, '.runtime/update')
journal = read(directory / 'journal.json', {})
if journal.get('phase') in (None, 'complete', 'rolledBack'):
return {'recovered': False}
if live(read(directory / 'worker.json', {})):
raise RuntimeError('更新正在进行,请等待原窗口完成。')
with (directory / 'worker.lock').open('a+b') as lock:
try:
if os.name == 'nt':
import msvcrt
lock.seek(0)
msvcrt.locking(lock.fileno(), msvcrt.LK_NBLCK, 1)
else:
import fcntl
fcntl.flock(lock.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
raise RuntimeError('已有更新恢复正在进行,请等待完成。') from None
identity = {'pid': os.getpid(), 'created': psutil.Process().create_time()}
worker = directory / 'worker.json'
temporary = worker.with_suffix('.bootstrap.tmp')
temporary.write_text(json.dumps(identity), encoding='utf-8')
temporary.replace(worker)
try:
journal = read(directory / 'journal.json', {})
if journal.get('phase') == 'installing':
restore_program_before_import(root, directory, journal)
elif journal.get('phase') == 'stopping':
stop_owned_services(root)
else:
raise RuntimeError('更新恢复阶段无效,未替换任何程序文件。')
# This import occurs only after restoring a complete, verified old
# program set; no mutable application module was imported above.
path = safe(root, 'deploy/offline/update_manager.py')
spec = importlib.util.spec_from_file_location('quality_recovered_updater', path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module.recover_locked(root)
finally:
if read(worker, {}) == identity:
worker.unlink(missing_ok=True)
if __name__ == '__main__':
try:
print(json.dumps(recover(), ensure_ascii=False))
except Exception as error:
print(json.dumps({'error': str(error) if isinstance(error, RuntimeError) else '更新恢复未完成,请保留本机备份并联系管理员。'}, ensure_ascii=False))
raise SystemExit(1)
+328
View File
@@ -0,0 +1,328 @@
#!/usr/bin/env python3
"""Real, isolated MySQL application-update acceptance on the macOS build host.
Never reads the source deployment configuration or connects to its MySQL port.
The test copies the application and creates its own database server and ports.
Only HTTP transport and the target-runtime verify action are adapted for the
host; signature verification, backup, file replacement, startup and rollback
are the production updater implementation.
"""
from __future__ import annotations
import argparse
import base64
from datetime import datetime, timezone
import hashlib
import http.cookiejar
import json
import os
from pathlib import Path
import secrets
import shutil
import socket
import subprocess
import sys
import tempfile
import time
import traceback
from unittest.mock import patch
import urllib.request
import zipfile
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
from cryptography.hazmat.primitives.serialization import Encoding, PublicFormat
import update_manager as update
SOURCE = Path(__file__).resolve().parents[2]
OPENER = urllib.request.build_opener(urllib.request.ProxyHandler({}))
def free_port():
with socket.socket() as test:
test.bind(('127.0.0.1', 0))
return test.getsockname()[1]
def backend_port():
for port in (5285, 5284, 5283):
try:
with socket.socket() as test:
test.bind(('127.0.0.1', port))
return port
except OSError:
continue
raise RuntimeError('No unused standalone test backend port is available; no existing instance was touched.')
def query(root, sql, database=None):
config = update.read(root / '.runtime/deployment.json')
command = [str(root / 'runtimes/mysql/bin/mysql'), '--no-defaults', '--protocol=TCP', '--host=127.0.0.1',
'--port=' + str(config['mysqlPort']), '--user=root', '--connect-timeout=5', '--batch', '--skip-column-names',
'--default-character-set=utf8mb4']
if database:
command.append(database)
env = os.environ.copy()
env['MYSQL_PWD'] = config['rootPassword']
result = subprocess.run(command, input=sql, encoding='utf-8', capture_output=True, env=env, timeout=60)
if result.returncode:
(root / '.runtime/update/test-sql-error.log').write_text(result.stderr)
raise RuntimeError('SQL failed in the isolated test server; see its private test log.')
return result.stdout.strip()
def web(root, path='/'):
config = update.read(root / '.runtime/deployment.json')
with OPENER.open(f'http://127.0.0.1:{config["webPort"]}' + path, timeout=10) as response:
return response.read()
def login(root, username, password):
config = update.read(root / '.runtime/deployment.json')
base = f'http://127.0.0.1:{config["webPort"]}'
cookies = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}), urllib.request.HTTPCookieProcessor(cookies))
with opener.open(base + '/api/auth/csrf', timeout=10) as response:
token = json.load(response)['token']
request = urllib.request.Request(base + '/api/auth/login', data=json.dumps({'username': username, 'password': password}).encode(),
headers={'Content-Type': 'application/json', 'X-XSRF-TOKEN': token})
with opener.open(request, timeout=15) as response:
assert response.status == 200
with opener.open(base + '/api/auth/me', timeout=10) as response:
assert response.status == 200
def prepare(work):
# MySQL's Unix socket has a 103-byte path limit on this host.
root = Path(tempfile.mkdtemp(prefix='qc-upd-', dir='/tmp')).resolve()
(work / 'isolated-root.txt').write_text(str(root) + '\n')
for name in ('app/web', 'scripts', 'deploy/offline', 'runtimes/python/bin', '.runtime/update',
'runtimedata/logs', 'runtimedata/sources', 'runtimedata/archive', 'backend/src/main/resources'):
(root / name).mkdir(parents=True, exist_ok=True)
for name in ('control.py', 'gateway.py', 'update_manager.py'):
shutil.copy2(SOURCE / 'deploy/offline' / name, root / 'deploy/offline' / name)
shutil.copy2(SOURCE / 'deploy/offline/runtime_platform.py', root / 'scripts/runtime_platform.py')
# No AI processes run in this acceptance test; preserve supervisor cleanup's
# normal subprocess path while keeping it confined to the disposable root.
(root / 'scripts/nao_quality_runtime.py').write_text('"""Isolated test: no AI processes have been started."""\n')
shutil.copy2(SOURCE / 'backend/target/quality-platform-backend-0.1.0-SNAPSHOT.jar', root / 'app/backend.jar')
shutil.copy2(SOURCE / 'backend/src/main/resources/analysis-runtime-policy.json', root / 'backend/src/main/resources/analysis-runtime-policy.json')
(root / 'app/web/index.html').write_text('<!doctype html><html><body><div id="root">isolated-update-v1</div></body></html>')
(root / 'app/web/obsolete.js').write_text('/* obsolete test application asset */')
(root / 'runtimes/java').symlink_to(SOURCE / '.runtime/jdk/Contents/Home', target_is_directory=True)
(root / 'runtimes/mysql').symlink_to(SOURCE / '.runtime/mysql', target_is_directory=True)
(root / 'runtimes/python/bin/python3').symlink_to(sys.executable)
key = Ed25519PrivateKey.generate()
(root / update.KEY).write_bytes(key.public_key().public_bytes(Encoding.PEM, PublicFormat.SubjectPublicKeyInfo))
config = {'mysqlPort': free_port(), 'backendPort': backend_port(), 'webPort': free_port(), 'webBind': '127.0.0.1',
'mysqlDatabase': 'quality_platform', 'archiveDirectory': 'runtimedata/archive',
'mysqlPassword': secrets.token_urlsafe(32), 'rootPassword': secrets.token_urlsafe(32),
'adminUsername': 'update-test-admin', 'adminPassword': secrets.token_urlsafe(24)}
assert config['mysqlPort'] != 53306
update.atomic(root / '.runtime/deployment.json', config)
(root / '.runtime/deployment.json').chmod(0o600)
private_marker = secrets.token_hex(20)
update.atomic(root / '.runtime/analysis-api-settings.json', {'testPrivateMarker': private_marker})
manifest = {'platform': 'darwin', 'architecture': 'arm64', 'updateVersion': 'host-initial', 'updateSequence': 1,
'runtimes': [{'name': 'java', 'hostTestRuntime': True}, {'name': 'mysql', 'hostTestRuntime': True},
{'name': 'python', 'version': sys.version.split()[0], 'hostTestRuntime': True}],
'pythonWheels': [], 'files': []}
for path in sorted(root.rglob('*')):
if path.is_file() and not path.is_symlink() and not path.relative_to(root).as_posix().startswith(('.runtime/', 'runtimedata/', 'runtimes/')):
manifest['files'].append({'path': path.relative_to(root).as_posix(), 'sha256': update.digest(path)})
update.atomic(root / 'manifest.json', manifest)
return root, key, config, private_marker
def release_fixture(root, work, key, version, sequence, bad=False):
folder = work / ('fixture-' + version)
folder.mkdir()
program = {}
for entry in update.read(root / 'manifest.json')['files']:
name = entry['path']
if update.program_path(name) and name != 'app/web/obsolete.js':
program[name] = root / name
index = folder / 'index.html'
index.write_text(f'<!doctype html><html><body><div id="root">isolated-update-{version}</div></body></html>')
program['app/web/index.html'] = index
if bad:
bad_jar = folder / 'bad-backend.jar'
# The real new Java process runs Flyway, changes data, then fails. This
# proves rollback restores the dump, rather than merely restarting Java.
migration = ("UPDATE platform_metadata SET metadata_value='FAILED-UPDATE-DATA' WHERE metadata_key='update_integration_sentinel';\n"
"UPDATE app_user SET password_hash='FAILED-UPDATE-HASH' WHERE username='update-test-admin';\n"
"CREATE TABLE isolated_update_failure_marker(id INT PRIMARY KEY);\n"
"SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT='Intentional isolated updater integration failure';\n")
with zipfile.ZipFile(root / 'app/backend.jar') as source, zipfile.ZipFile(bad_jar, 'w') as target:
for member in source.infolist():
target.writestr(member, source.read(member))
target.writestr('BOOT-INF/classes/db/migration/V999__isolated_update_failure.sql', migration)
program['app/backend.jar'] = bad_jar
archive = folder / ('quality-update-' + version + '.zip')
with zipfile.ZipFile(archive, 'w', zipfile.ZIP_DEFLATED, compresslevel=1) as target:
for name, path in sorted(program.items()):
target.write(path, name)
release = {'schema': 1, 'platform': 'windows-x64', 'version': version, 'sequence': sequence,
'runtimeFingerprint': update.runtime_fingerprint(update.read(root / 'manifest.json')),
'archive': {'file': archive.name, 'size': archive.stat().st_size, 'sha256': update.digest(archive)},
'files': [{'path': name, 'size': path.stat().st_size, 'sha256': update.digest(path)} for name, path in sorted(program.items())]}
payload = update.canonical(release)
envelope = {'payload': base64.b64encode(payload).decode(), 'signature': base64.b64encode(key.sign(payload)).decode()}
(folder / 'latest.json').write_bytes(update.canonical(envelope))
return release, envelope, archive
def run_update(root, release, envelope, archive):
real_controller = update.controller
verify_checks = []
def host_controller(target, action, database_only=False, timeout=1100):
assert target == root
if action != 'verify':
return real_controller(target, action, database_only, timeout)
# Host adaptation: this ARM Mac cannot satisfy Windows/x64 runtime probes.
# All installed program hashes and the native Java/MySQL executables are
# still checked. Real full application startup is checked below by serve.
for entry in update.read(root / 'manifest.json')['files']:
assert update.digest(root / entry['path']) == entry['sha256'], entry['path']
for command in ([root / 'runtimes/java/bin/java', '-version'],
[root / 'runtimes/mysql/bin/mysqld', '--no-defaults', '--version']):
result = subprocess.run([str(a) for a in command], capture_output=True, timeout=30)
assert result.returncode == 0
verify_checks.append(True)
def transport(url, target=None, expected_size=None, expected_hash=None):
if url == update.LATEST and target is None:
return update.canonical(envelope)
assert url == update.BASE + archive.name and target is not None
assert expected_size == archive.stat().st_size and expected_hash == update.digest(archive)
with archive.open('rb') as reader, target.open('xb') as writer:
shutil.copyfileobj(reader, writer)
return b''
launcher = subprocess.Popen([sys.executable, '-c', 'pass'])
launcher.wait(timeout=10)
token = secrets.token_hex(16)
update.atomic(root / '.runtime/update/request.json', {'token': token, 'created': time.time()})
# Patch only this test process. Production Windows/source enablement remains
# unchanged, including in the real backend processes started by control.py.
with patch.object(update, 'enabled', return_value=True), patch.object(update, 'download', side_effect=transport), \
patch.object(update, 'controller', side_effect=host_controller):
update.worker(launcher.pid, token, root)
assert verify_checks, update.read(root / '.runtime/update/status.json').get('message')
return update.read(root / '.runtime/update/status.json')
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--work', type=Path, required=True)
parser.add_argument('--expected-tables', type=int, default=83)
args = parser.parse_args()
work = args.work.resolve()
work.mkdir(parents=True, exist_ok=False)
root = None
report = {'startedAt': datetime.now(timezone.utc).isoformat(), 'passed': False,
'environment': 'macOS ARM host, disposable native MySQL + Java + Python runtimes',
'productionServicesTouched': False, 'productionDatabasePortUsed': False,
'networkTransport': 'local signed fixtures substituted only in test process',
'targetRuntimeVerify': 'host adapter: all manifest hashes plus native Java/MySQL probes; actual backend restarts',
'windowsNativeProcessDetachmentValidated': False, 'windowsNativeRuntimeCompatibilityValidated': False,
'checks': {}}
try:
print('Preparing isolated update acceptance root.', flush=True)
root, key, config, private_marker = prepare(work)
report['isolatedRoot'] = str(root)
report['ports'] = {k: config[k] for k in ('mysqlPort', 'backendPort', 'webPort')}
print('Starting isolated MySQL and Java; applying real Flyway schema.', flush=True)
process = update.controller(root, 'serve')
update.wait_ready(root, process)
database = config['mysqlDatabase']
count = int(query(root, f"SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='{database}';"))
assert count == args.expected_tables, f'Expected {args.expected_tables} tables; actual isolated schema has {count}.'
report['checks']['tableCount'] = count
sentinel = secrets.token_hex(24)
query(root, f"INSERT INTO platform_metadata(metadata_key,metadata_value) VALUES ('update_integration_sentinel','{sentinel}');", database)
account_hash = query(root, "SELECT password_hash FROM app_user WHERE username='update-test-admin';", database)
assert account_hash.startswith('$2')
login(root, config['adminUsername'], config['adminPassword'])
report['checks']['initialLogin'] = True
config['restoredFromMigration'] = 'isolated-update-migration-fixture'
update.atomic(root / '.runtime/deployment.json', config)
update.atomic(root / '.runtime/migration-receipt.json', {'migrationId': config['restoredFromMigration'],
'database': database, 'migrationManifestSha256': 'a' * 64})
before_pid = update.read(root / '.runtime/service.json')['supervisor']['pid']
print('Applying signed success release with actual dump, file replacement and service restart.', flush=True)
good = release_fixture(root, work, key, 'host-success', 2)
result = run_update(root, *good)
assert result['state'] == 'completed', result.get('message')
assert update.read(root / 'manifest.json')['updateVersion'] == 'host-success'
assert b'isolated-update-host-success' in web(root)
assert not (root / 'app/web/obsolete.js').exists()
assert update.read(root / '.runtime/service.json')['supervisor']['pid'] != before_pid
assert query(root, "SELECT metadata_value FROM platform_metadata WHERE metadata_key='update_integration_sentinel';", database) == sentinel
assert query(root, "SELECT password_hash FROM app_user WHERE username='update-test-admin';", database) == account_hash
login(root, config['adminUsername'], config['adminPassword'])
first_journal = update.read(root / '.runtime/update/journal.json')
first_dump = root / '.runtime/update' / first_journal['backup'] / 'database.sql'
assert first_dump.is_file() and first_dump.stat().st_size > 10000
report['checks'].update(realMysqldump=True, successProgramSwitch=True, realServiceRestart=True,
obsoleteProgramRemoved=True, successSentinelPreserved=True,
successAccountHashPreserved=True, successLogin=True,
successDumpBytes=first_dump.stat().st_size)
print('Applying signed bad Java release: real Flyway mutates sentinel/hash then intentionally fails.', flush=True)
bad = release_fixture(root, work, key, 'host-bad', 3, bad=True)
result = run_update(root, *bad)
assert result['state'] == 'failed' and '已恢复原版本' in result['message'], result
restored = update.read(root / '.runtime/deployment.json')
restored_db = restored['mysqlDatabase']
assert restored_db != database and restored_db.startswith('quality_restore_')
assert update.read(root / 'manifest.json')['updateVersion'] == 'host-success'
assert b'isolated-update-host-success' in web(root)
assert query(root, "SELECT metadata_value FROM platform_metadata WHERE metadata_key='update_integration_sentinel';", restored_db) == sentinel
assert query(root, "SELECT password_hash FROM app_user WHERE username='update-test-admin';", restored_db) == account_hash
assert int(query(root, f"SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='{restored_db}';")) == count
assert query(root, "SELECT metadata_value FROM platform_metadata WHERE metadata_key='update_integration_sentinel';", database) == 'FAILED-UPDATE-DATA', 'Failed original sentinel did not retain the migration write.'
assert query(root, "SELECT password_hash FROM app_user WHERE username='update-test-admin';", database) == 'FAILED-UPDATE-HASH', 'Failed original account hash did not retain the migration write.'
assert query(root, "SELECT COUNT(*) FROM flyway_schema_history WHERE version='999' AND success=0;", database) == '1'
assert query(root, "SELECT COUNT(*) FROM flyway_schema_history WHERE version='999';", restored_db) == '0'
assert update.read(root / '.runtime/analysis-api-settings.json')['testPrivateMarker'] == private_marker
assert restored['archiveDirectory'] == config['archiveDirectory']
assert restored['rootPassword'] == config['rootPassword'] and restored['mysqlPassword'] == config['mysqlPassword']
assert update.read(root / '.runtime/migration-receipt.json')['database'] == restored_db
login(root, config['adminUsername'], config['adminPassword'])
failed_journal = update.read(root / '.runtime/update/journal.json')
assert failed_journal['phase'] == 'rolledBack'
failed_dump = root / '.runtime/update' / failed_journal['backup'] / 'database.sql'
report['checks'].update(badReleaseActuallyMutatedDatabase=True, failedOriginalDatabasePreserved=True,
restoredIntoNewDatabase=True, restoredTableCount=count, rollbackSentinelRestored=True,
rollbackAccountHashRestored=True, rollbackLogin=True, oldProgramRestarted=True,
privateConfigurationPreserved=True, archiveDirectoryPreserved=True,
migrationReceiptDatabaseAligned=True,
failedFlywayHistoryPreserved=True, restoredFlywayHistoryClean=True,
rollbackDumpBytes=failed_dump.stat().st_size,
originalDatabase=database, rollbackDatabase=restored_db)
report['passed'] = True
print('Real MySQL success and failed-migration rollback acceptance passed.', flush=True)
except Exception as error:
report['error'] = str(error)
(work / 'test-failure-trace.txt').write_text(traceback.format_exc())
print('Isolated acceptance failed: ' + str(error), flush=True)
finally:
if root is not None:
try:
update.controller(root, 'stop', timeout=180)
state = update.read(root / '.runtime/service.json', {})
assert not update.live(state.get('supervisor', {}))
report['isolatedServicesStopped'] = True
except Exception as error:
report['isolatedServicesStopped'] = False
report['cleanupError'] = str(error)
report['passed'] = False
report['finishedAt'] = datetime.now(timezone.utc).isoformat()
(work / 'report.json').write_text(json.dumps(report, ensure_ascii=False, indent=2) + '\n')
print(json.dumps(report, ensure_ascii=False), flush=True)
return 0 if report['passed'] else 1
if __name__ == '__main__':
raise SystemExit(main())
+646
View File
@@ -0,0 +1,646 @@
#!/usr/bin/env python3
"""Signed Windows standalone application updates. No caller-supplied URLs or paths.
The release envelope signs the *decoded payload bytes* with Ed25519. Runtime
identity is SHA256(canonical({runtimes, pythonWheels})). Program files are a full
snapshot; private data and runtimes are never update destinations.
"""
from __future__ import annotations
import argparse
import base64
import contextlib
from datetime import datetime, timezone
import hashlib
import json
import os
from pathlib import Path, PurePosixPath
import re
import secrets
import shutil
import stat
import subprocess
import sys
import time
import zipfile
from urllib.request import ProxyHandler, HTTPRedirectHandler, build_opener
import psutil
from cryptography.hazmat.primitives.serialization import load_pem_public_key
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey
ROOT = Path(__file__).resolve().parents[2]
BASE = 'https://quality.lyunlab.me/quality-updates/windows-x64/'
LATEST = BASE + 'latest.json'
KEY = 'deploy/offline/update-public-key.pem'
MAX_MANIFEST = 16 * 1024 * 1024
MAX_ARCHIVE = 2 * 1024**3
MAX_EXPANDED = 4 * 1024**3
ACTIVE = {'queued', 'downloading', 'backingUp', 'installing', 'restarting', 'rollingBack'}
SHA = re.compile(r'[a-f0-9]{64}')
class UpdateError(RuntimeError):
pass
class BusyUpdate(UpdateError):
pass
def canonical(value):
return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(',', ':')).encode('utf-8')
def unique_object(pairs):
value = {}
for key, item in pairs:
if key in value:
raise UpdateError('更新清单包含重复字段。')
value[key] = item
return value
def decode(raw):
return json.loads(raw, object_pairs_hook=unique_object)
def digest(path):
with path.open('rb') as stream:
return hashlib.file_digest(stream, 'sha256').hexdigest()
def atomic(path, value):
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
temporary = path.with_name(path.name + '.' + secrets.token_hex(6) + '.tmp')
with temporary.open('xb') as stream:
stream.write(canonical(value))
stream.flush()
os.fsync(stream.fileno())
temporary.replace(path)
def copy_atomic(source, target):
target.parent.mkdir(parents=True, exist_ok=True)
temporary = target.with_name(target.name + '.update-' + secrets.token_hex(6))
try:
with source.open('rb') as reader, temporary.open('xb') as writer:
shutil.copyfileobj(reader, writer)
writer.flush()
os.fsync(writer.fileno())
shutil.copystat(source, temporary)
temporary.replace(target)
finally:
temporary.unlink(missing_ok=True)
def read(path, default=None):
return decode(path.read_bytes()) if path.exists() else default
def runtime_fingerprint(manifest):
return hashlib.sha256(canonical({key: manifest.get(key, []) for key in ('runtimes', 'pythonWheels')})).hexdigest()
def safe_path(name):
if (not isinstance(name, str) or not name or len(name) > 230 or '\\' in name
or ':' in name or '\x00' in name or name.startswith('/')
or any(ord(c) < 32 for c in name)):
raise UpdateError('更新文件路径无效。')
parts = name.split('/')
if any(p in ('', '.', '..') or p.endswith((' ', '.'))
or re.fullmatch(r'(?i)(con|prn|aux|nul|com[1-9]|lpt[1-9])(?:\..*)?', p) for p in parts):
raise UpdateError('更新文件路径无效。')
return PurePosixPath(name)
def program_path(name):
path = safe_path(name)
if name == KEY or name.startswith('deploy/bootstrap/'):
return False # immutable trust anchor, even in a correctly signed release
return (name == 'app/backend.jar' or name.startswith('app/web/')
or name == 'backend/src/main/resources/analysis-runtime-policy.json'
or name.startswith('scripts/') and path.suffix == '.py'
or name.startswith('ai/nao-quality/')
or name.startswith('deploy/offline/') and path.suffix in ('.py', '.ps1', '.md'))
def destination(root, name):
safe_path(name)
target = root.joinpath(*name.split('/'))
for path in (target, *target.parents):
if path == root:
break
if path.is_symlink():
raise UpdateError('更新目录包含符号链接,已停止更新。')
if os.name == 'nt' and path.exists() and getattr(path.lstat(), 'st_file_attributes', 0) & stat.FILE_ATTRIBUTE_REPARSE_POINT:
raise UpdateError('更新目录包含重解析点,已停止更新。')
if not target.resolve().is_relative_to(root.resolve()):
raise UpdateError('更新文件超出程序目录。')
return target
def entry_map(entries, program_only=False):
if not isinstance(entries, list) or len(entries) > 50000:
raise UpdateError('更新文件清单无效。')
result, folded = {}, set()
for entry in entries:
if not isinstance(entry, dict):
raise UpdateError('更新文件清单无效。')
name = entry.get('path')
safe_path(name)
if name.casefold() in folded:
raise UpdateError('更新文件路径重复。')
folded.add(name.casefold())
if program_only and not program_path(name):
raise UpdateError('更新包含不允许替换的文件。')
if entry.get('symlink') is not None or not SHA.fullmatch(str(entry.get('sha256', ''))):
raise UpdateError('更新文件校验信息无效。')
result[name] = entry
return result
def validate_release(envelope, root=ROOT):
try:
raw = base64.b64decode(envelope['payload'], validate=True)
signature = base64.b64decode(envelope['signature'], validate=True)
key = load_pem_public_key(destination(root, KEY).read_bytes())
if not isinstance(key, Ed25519PublicKey):
raise ValueError('key type')
key.verify(signature, raw)
release = decode(raw)
except Exception as error:
raise UpdateError('更新签名验证失败。') from error
current = read(root / 'manifest.json')
if release.get('schema') != 1 or release.get('platform') != 'windows-x64':
raise UpdateError('更新包不适用于此系统。')
if not isinstance(release.get('version'), str) or not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9._-]{0,79}', release['version']):
raise UpdateError('更新版本无效。')
if type(release.get('sequence')) is not int or release['sequence'] <= 0:
raise UpdateError('更新版本序号无效。')
if release.get('runtimeFingerprint') != runtime_fingerprint(current):
raise UpdateError('运行环境版本不同,请使用完整安装包更新。')
archive = release.get('archive', {})
if (not re.fullmatch(r'quality-update-[A-Za-z0-9][A-Za-z0-9._-]{0,100}\.zip', str(archive.get('file', '')))
or type(archive.get('size')) is not int or not 0 < archive['size'] <= MAX_ARCHIVE
or not SHA.fullmatch(str(archive.get('sha256', '')))):
raise UpdateError('更新下载信息无效。')
files = entry_map(release.get('files'), True)
if not files or any(type(e.get('size')) is not int or e['size'] < 0 for e in files.values()) or sum(e['size'] for e in files.values()) > MAX_EXPANDED:
raise UpdateError('更新文件大小无效。')
if 'bundleManifest' in release:
raise UpdateError('更新清单不能包含部署数据。')
old_files = entry_map(current.get('files'))
required = {'app/backend.jar', 'deploy/offline/control.py', 'deploy/offline/update_manager.py',
'backend/src/main/resources/analysis-runtime-policy.json'}
if not required.issubset(files) or not any(name.startswith('app/web/') for name in files):
raise UpdateError('更新缺少必要程序文件。')
if KEY not in old_files or digest(destination(root, KEY)) != old_files[KEY]['sha256']:
raise UpdateError('更新信任公钥校验失败。')
for name in set(old_files) | set(files):
destination(root, name)
return release
class NoRedirect(HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
raise UpdateError('更新服务器返回了不允许的跳转。')
def download(url, target=None, expected_size=None, expected_hash=None):
if url != LATEST and not (url.startswith(BASE) and re.fullmatch(r'quality-update-[A-Za-z0-9][A-Za-z0-9._-]{0,100}\.zip', url[len(BASE):])):
raise UpdateError('更新下载地址无效。')
limit = expected_size if expected_size is not None else MAX_MANIFEST
hasher, count, chunks = hashlib.sha256(), 0, []
with build_opener(ProxyHandler({}), NoRedirect()).open(url, timeout=30) as response:
if response.geturl() != url or response.status != 200:
raise UpdateError('更新服务器响应无效。')
with target.open('xb') if target is not None else contextlib.nullcontext() as stream:
while block := response.read(1024 * 1024):
count += len(block)
if count > limit:
raise UpdateError('更新下载大小不符合清单。')
hasher.update(block)
if stream is not None:
stream.write(block)
else:
chunks.append(block)
if expected_size is not None and (count != expected_size or hasher.hexdigest() != expected_hash):
raise UpdateError('更新下载校验失败。')
return b''.join(chunks)
def unpack(archive, stage, release):
entries = {e['path']: e for e in release['files']}
with zipfile.ZipFile(archive) as source:
seen = set()
for member in source.infolist():
name = member.filename
if name not in entries or name in seen or member.is_dir() or member.flag_bits & 1:
raise UpdateError('更新压缩包文件清单不符。')
seen.add(name)
if stat.S_ISLNK(member.external_attr >> 16) or member.file_size != entries[name]['size']:
raise UpdateError('更新压缩包文件类型或大小不符。')
target = destination(stage, name)
target.parent.mkdir(parents=True, exist_ok=True)
count = 0
with source.open(member) as reader, target.open('xb') as writer:
while block := reader.read(1024 * 1024):
count += len(block)
if count > entries[name]['size']:
raise UpdateError('更新解压大小不符。')
writer.write(block)
if digest(target) != entries[name]['sha256']:
raise UpdateError('更新解压校验失败。')
if seen != set(entries):
raise UpdateError('更新压缩包缺少文件。')
def enabled(root=ROOT):
return os.name == 'nt' and os.environ.get('QUALITY_UPDATE_ENABLED') == 'true' and (root / 'runtimes/python/python.exe').is_file() and (root / 'manifest.json').is_file()
def status(root=ROOT):
manifest = read(root / 'manifest.json', {})
value = read(root / '.runtime/update/status.json', {})
allowed = {'state', 'message', 'checkedAt', 'version', 'available', 'startedAt', 'finishedAt'}
result = {k: v for k, v in value.items() if k in allowed}
result.update(enabled=enabled(root), currentVersion=manifest.get('updateVersion', manifest.get('builtAtUtc', '当前版本')))
result.setdefault('state', 'idle')
if result['state'] in ACTIVE and not live(read(root / '.runtime/update/worker.json', {})):
pending = read(root / '.runtime/update/request.json', {})
journal = read(root / '.runtime/update/journal.json', {})
if time.time() - pending.get('created', 0) > 120:
result.update(state='failed', available=False, message='更新进程已中断,请重新运行软件启动入口恢复。'
if journal.get('phase') not in (None, 'complete', 'rolledBack') else '更新进程已结束,请重新检查更新。')
if not result['enabled']:
result['message'] = '此部署不支持软件内更新。'
return result
def write_status(state, message, root=ROOT, **extra):
destination(root, '.runtime/update/status.json')
value = read(root / '.runtime/update/status.json', {})
value.update(extra, state=state, message=message)
atomic(root / '.runtime/update/status.json', value)
return status(root)
def check(root=ROOT):
release = validate_release(decode(download(LATEST)), root)
available = release['sequence'] > read(root / 'manifest.json').get('updateSequence', 0)
atomic(root / '.runtime/update/checked.json', release)
return write_status('available' if available else 'current', '发现新版本。' if available else '已是最新版本。', root,
version=release['version'], available=available, checkedAt=datetime.now(timezone.utc).isoformat())
def process_options():
return {'creationflags': subprocess.DETACHED_PROCESS | subprocess.CREATE_NEW_PROCESS_GROUP | subprocess.CREATE_BREAKAWAY_FROM_JOB} if os.name == 'nt' else {'start_new_session': True}
@contextlib.contextmanager
def exclusive_update(root=ROOT):
directory = root / '.runtime/update'
destination(root, '.runtime/update')
directory.mkdir(parents=True, exist_ok=True)
if live(read(directory / 'worker.json', {})):
raise BusyUpdate('已有更新或恢复正在进行。')
sys.path.insert(0, str(root / 'scripts'))
from runtime_platform import fcntl
with (directory / 'worker.lock').open('a+b') as lock:
try:
fcntl.flock(lock.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
except (BlockingIOError, OSError):
raise BusyUpdate('已有更新或恢复正在进行。') from None
identity = {'pid': os.getpid(), 'created': psutil.Process().create_time()}
atomic(directory / 'worker.json', identity)
try:
yield
finally:
if read(directory / 'worker.json', {}) == identity:
(directory / 'worker.json').unlink(missing_ok=True)
def launch(root=ROOT):
with exclusive_update(root):
return launch_locked(root)
def launch_locked(root=ROOT):
directory = root / '.runtime/update'
destination(root, '.runtime/update')
directory.mkdir(parents=True, exist_ok=True)
# This short launcher MUST exit before its worker may stop Java. The worker
# verifies launcher disappearance; DETACHED alone does not remove ancestry.
pending = read(directory / 'request.json', {})
if pending and time.time() - pending.get('created', 0) < 120:
raise BusyUpdate('已有更新正在启动。')
check(root) # refetch and verify; do not trust browser or cached JSON
if not status(root).get('available'):
return status(root)
token = secrets.token_hex(16)
atomic(directory / 'request.json', {'token': token, 'created': time.time()})
write_status('queued', '更新已开始,完成后将自动重新连接。', root, startedAt=datetime.now(timezone.utc).isoformat(), available=False)
try:
with (directory / 'worker.log').open('ab') as log:
subprocess.Popen([sys.executable, str(Path(__file__).resolve()), 'worker', '--launcher', str(os.getpid()), '--token', token],
cwd=root, env=os.environ.copy(), stdin=subprocess.DEVNULL, stdout=log, stderr=log, **process_options())
except Exception:
(directory / 'request.json').unlink(missing_ok=True)
raise UpdateError('独立更新进程无法启动,服务仍在运行。') from None
return status(root)
def live(info):
try:
process = psutil.Process(info['pid'])
return process.is_running() and process.status() != psutil.STATUS_ZOMBIE and abs(process.create_time() - info['created']) < .01
except (psutil.Error, KeyError, TypeError):
return False
def controller(root, action, database_only=False, timeout=1100):
python = root / 'runtimes/python' / ('python.exe' if os.name == 'nt' else 'bin/python3')
command = [str(python), str(root / 'deploy/offline/control.py'), action]
if database_only:
command += ['--database-only']
if action == 'serve':
with (root / '.runtime/update/service.log').open('ab') as log:
return subprocess.Popen(command, cwd=root, env=os.environ.copy(), stdin=subprocess.DEVNULL, stdout=log, stderr=log, **process_options())
result = subprocess.run(command, cwd=root, env=os.environ.copy(), stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout)
if result.returncode:
raise UpdateError('服务启停失败,请查看本机更新日志。')
return result
def wait_ready(root, process, database_only=False):
deadline = time.monotonic() + (180 if database_only else 1000)
while time.monotonic() < deadline:
state = read(root / '.runtime/service.json', {})
if state.get('ready') and state.get('databaseOnly', False) == database_only and state.get('supervisor', {}).get('pid') == process.pid:
return
if process.poll() is not None:
raise UpdateError('更新后服务启动失败,正在恢复旧版本。')
time.sleep(.5)
raise UpdateError('更新后服务启动超时,正在恢复旧版本。')
def mysql(root, config, binary='mysql', database=None, stdin=None, stdout=subprocess.PIPE):
exe = '.exe' if os.name == 'nt' else ''
command = [str(root / 'runtimes/mysql/bin' / (binary + exe)), '--no-defaults', '--protocol=TCP', '--host=127.0.0.1',
'--port=' + str(config['mysqlPort']), '--user=root', '--default-character-set=utf8mb4']
if binary == 'mysqldump':
command += ['--single-transaction', '--routines', '--events', '--triggers', '--hex-blob', '--no-tablespaces', '--set-gtid-purged=OFF', config.get('mysqlDatabase', 'quality_platform')]
elif database:
command.append(database)
if binary == 'mysql':
command.append('--connect-timeout=10')
env = os.environ.copy()
env['MYSQL_PWD'] = config['rootPassword']
result = subprocess.run(command, cwd=root, env=env, stdin=stdin, stdout=stdout, stderr=subprocess.PIPE, timeout=3600)
if result.returncode:
error_log = destination(root, '.runtime/update/database-error.log')
with error_log.open('ab') as log:
log.write((binary + ' failed\n').encode() + result.stderr[-16384:] + b'\n')
raise UpdateError('数据库备份或恢复失败,已保留本机备份和原数据库。')
def sql(root, config, query):
# stdin must be a real file on Windows; do not place passwords in arguments.
path = root / '.runtime/update/query.sql'
try:
path.write_text(query, encoding='utf-8')
with path.open('rb') as source:
mysql(root, config, stdin=source)
finally:
path.unlink(missing_ok=True)
def backup_program(root, backup, release):
current = read(root / 'manifest.json')
names = sorted({e['path'] for e in current['files'] if program_path(e['path'])} | {e['path'] for e in release['files']})
present = []
for name in names:
source = destination(root, name)
if source.exists():
target = destination(backup / 'program', name)
target.parent.mkdir(parents=True, exist_ok=True)
copy_atomic(source, target)
present.append(name)
copy_atomic(root / 'manifest.json', backup / 'manifest.json')
# All private configuration (AI keys, deployment credentials and bootstrap
# flags) is backed up locally, but never becomes a release payload.
shutil.copytree(root / '.runtime', backup / 'private', ignore=shutil.ignore_patterns('update', '*.lock', '*.log', 'service.json', 'stop-request.json'), symlinks=False)
for path in (backup / 'private').rglob('*'):
if path.is_file():
with path.open('rb') as stream:
os.fsync(stream.fileno())
atomic(backup / 'program.json', {'all': names, 'present': present})
def replace_program(root, stage, backup, release):
current = read(root / 'manifest.json')
names = read(backup / 'program.json')['all']
wanted = {e['path'] for e in release['files']}
for name in names:
target = destination(root, name)
if name in wanted:
copy_atomic(stage / name, target)
else:
target.unlink(missing_ok=True)
current['files'] = ([e for e in current['files'] if not program_path(e['path'])]
+ [{'path': e['path'], 'sha256': e['sha256']} for e in release['files']])
current.update(updateSequence=release['sequence'], updateVersion=release['version'])
atomic(root / 'manifest.json', current)
def restore_program(root, backup):
names = read(backup / 'program.json')
for name in names['all']:
target = destination(root, name)
if name in names['present']:
copy_atomic(backup / 'program' / name, target)
else:
target.unlink(missing_ok=True)
copy_atomic(backup / 'manifest.json', destination(root, 'manifest.json'))
for source in (backup / 'private').rglob('*'):
if source.is_file():
relative = source.relative_to(backup / 'private').as_posix()
copy_atomic(source, destination(root, '.runtime/' + relative))
def rollback(root, backup, database_backed_up):
write_status('rollingBack', '更新未完成,正在恢复原版本。', root)
controller(root, 'stop', timeout=180)
restore_program(root, backup)
if database_backed_up:
config = read(root / '.runtime/deployment.json')
process = controller(root, 'serve', True)
wait_ready(root, process, True)
database = 'quality_restore_' + secrets.token_hex(12)
sql(root, config, f'CREATE DATABASE `{database}` CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;')
with (backup / 'database.sql').open('rb') as source:
mysql(root, config, database=database, stdin=source)
sql(root, config, f"GRANT ALL PRIVILEGES ON `{database}`.* TO 'quality_app'@'127.0.0.1';")
controller(root, 'stop', timeout=180)
config['mysqlDatabase'] = database
atomic(root / '.runtime/deployment.json', config)
receipt_path = root / '.runtime/migration-receipt.json'
receipt = read(receipt_path, {})
if receipt and receipt.get('migrationId') == config.get('restoredFromMigration'):
receipt['database'] = database
atomic(receipt_path, receipt)
process = controller(root, 'serve')
wait_ready(root, process)
def worker(launcher, token, root=ROOT):
directory = root / '.runtime/update'
for _ in range(300):
if not psutil.pid_exists(launcher):
break
time.sleep(.1)
else:
raise UpdateError('更新启动器未退出,已安全停止。')
if any('java' in p.name().lower() for p in psutil.Process().parents()):
raise UpdateError('更新进程未独立启动,已安全停止。')
if read(directory / 'request.json', {}).get('token') != token:
raise UpdateError('更新请求已失效。')
with exclusive_update(root):
backup = directory / ('backup-' + datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ') + '-' + secrets.token_hex(4))
stopped = backed_up = program_saved = changed = False
try:
envelope = decode(download(LATEST))
release = validate_release(envelope, root)
if release['sequence'] <= read(root / 'manifest.json').get('updateSequence', 0):
raise UpdateError('此更新版本已安装。')
backup.mkdir()
archive, stage = backup / 'update.zip', backup / 'stage'
expanded = sum(entry['size'] for entry in release['files'])
# Keep enough headroom for extraction, old program files and a
# logical dump; the dump itself must complete before replacement.
db_size = sum(p.stat().st_size for p in (root / 'runtimedata/mysql').rglob('*') if p.is_file())
old_program_size = sum((root / entry['path']).stat().st_size for entry in read(root / 'manifest.json')['files'] if program_path(entry['path']) and (root / entry['path']).is_file())
private_size = sum(p.stat().st_size for p in (root / '.runtime').rglob('*') if p.is_file() and not p.is_relative_to(directory))
if shutil.disk_usage(root).free < release['archive']['size'] + expanded + old_program_size + private_size + db_size * 3 + 512 * 1024**2:
raise UpdateError('磁盘剩余空间不足,请释放空间后重试。')
write_status('downloading', '正在下载并校验更新。', root)
info = release['archive']
download(BASE + info['file'], archive, info['size'], info['sha256'])
unpack(archive, stage, release)
# Revalidate immediately before any service or disk mutation.
validate_release(envelope, root)
write_status('backingUp', '正在备份,服务即将重新启动。', root)
atomic(directory / 'journal.json', {'backup': backup.name, 'phase': 'stopping', 'databaseBackedUp': False})
controller(root, 'stop', timeout=180)
stopped = True
backup_program(root, backup, release)
program_saved = True
config = read(root / '.runtime/deployment.json')
database = controller(root, 'serve', True)
wait_ready(root, database, True)
with (backup / 'database.sql').open('xb') as output:
mysql(root, config, 'mysqldump', stdout=output)
output.flush()
os.fsync(output.fileno())
if (backup / 'database.sql').stat().st_size < 100:
raise UpdateError('数据库备份为空,已停止更新。')
backed_up = True
controller(root, 'stop', timeout=180)
# The actual dump can exceed the source's physical storage. Do not
# replace anything unless a separate rollback database still fits.
required_recovery = max(db_size, (backup / 'database.sql').stat().st_size * 3) + 512 * 1024**2
if shutil.disk_usage(root).free < required_recovery:
raise UpdateError('备份已完成,但剩余磁盘空间不足以安全回退,已停止更新。')
atomic(directory / 'journal.json', {'backup': backup.name, 'phase': 'installing', 'databaseBackedUp': True,
'backupManifestSha256': digest(backup / 'manifest.json'),
'programInventorySha256': digest(backup / 'program.json')})
write_status('installing', '正在安装更新。', root)
changed = True
replace_program(root, stage, backup, release)
controller(root, 'verify', timeout=180)
write_status('restarting', '正在重新启动,请稍候。', root)
process = controller(root, 'serve')
wait_ready(root, process)
atomic(directory / 'journal.json', {'backup': backup.name, 'phase': 'complete', 'databaseBackedUp': True})
write_status('completed', '更新完成。', root, version=release['version'], finishedAt=datetime.now(timezone.utc).isoformat(), available=False)
except Exception as error:
message = str(error) if isinstance(error, UpdateError) else '更新未完成,请查看本机更新日志。'
if stopped:
try:
if program_saved and changed:
rollback(root, backup, backed_up)
else:
controller(root, 'stop', timeout=180)
process = controller(root, 'serve')
wait_ready(root, process)
atomic(directory / 'journal.json', {'backup': backup.name, 'phase': 'rolledBack', 'databaseBackedUp': backed_up})
message += ' 已恢复原版本。'
except Exception:
message += ' 自动恢复未完成,请联系管理员使用本机更新备份恢复。'
write_status('failed', message, root, available=False, finishedAt=datetime.now(timezone.utc).isoformat())
finally:
(directory / 'request.json').unlink(missing_ok=True)
def recover(root=ROOT):
with exclusive_update(root):
return recover_locked(root)
def recover_locked(root=ROOT):
directory = root / '.runtime/update'
journal = read(directory / 'journal.json', {})
if journal.get('phase') in (None, 'complete', 'rolledBack'):
return status(root)
name = journal.get('backup', '')
if not re.fullmatch(r'backup-[0-9]{8}T[0-9]{6}Z-[a-f0-9]{8}', name):
raise UpdateError('更新恢复记录无效。')
backup = directory / name
if journal.get('phase') == 'installing':
if (digest(backup / 'manifest.json') != journal.get('backupManifestSha256')
or digest(backup / 'program.json') != journal.get('programInventorySha256')):
raise UpdateError('更新恢复备份校验失败,已保留现场。')
rollback(root, backup, journal.get('databaseBackedUp') is True)
else:
controller(root, 'stop', timeout=180)
process = controller(root, 'serve')
wait_ready(root, process)
atomic(directory / 'journal.json', {**journal, 'phase': 'rolledBack'})
return write_status('failed', '上次更新中断,已恢复原版本。', root, available=False)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('action', choices=('status', 'check', 'launch', 'worker', 'recover'))
parser.add_argument('--launcher', type=int)
parser.add_argument('--token')
args = parser.parse_args()
if args.action == 'status':
return status()
if not enabled():
raise UpdateError('此部署不支持软件内更新。')
if args.action == 'worker':
if not args.launcher or not re.fullmatch(r'[a-f0-9]{32}', args.token or ''):
raise UpdateError('更新请求无效。')
worker(args.launcher, args.token)
return status()
if args.action == 'recover':
return recover()
journal = read(ROOT / '.runtime/update/journal.json', {})
if journal.get('phase') not in (None, 'complete', 'rolledBack'):
raise UpdateError('上次更新中断,请重新运行软件启动入口恢复。')
if live(read(ROOT / '.runtime/update/worker.json', {})):
raise BusyUpdate('已有更新正在进行。')
return check() if args.action == 'check' else launch()
if __name__ == '__main__':
try:
print(json.dumps(main(), ensure_ascii=False))
except Exception as error:
message = str(error) if isinstance(error, UpdateError) else '更新服务暂不可用,请稍后重试。'
if enabled() and not isinstance(error, BusyUpdate) and sys.argv[1:2] != ['status'] and not live(read(ROOT / '.runtime/update/worker.json', {})):
write_status('failed', message)
print(json.dumps({'error': message}, ensure_ascii=False))
raise SystemExit(1)
+7
View File
@@ -0,0 +1,7 @@
param([ValidateRange(1024,65535)][int]$Port = 5280)
$ErrorActionPreference = 'Stop'
Write-Host '此兼容脚本不执行任何系统级准备操作,也不需要管理员权限。'
Write-Host '运行库随离线包提供。请使用 windows-start.ps1 启动应用。'
Write-Host "网页端口为 $Port;其他设备能否访问取决于 Windows 当前防火墙和网络策略。"
exit 0
+64
View File
@@ -0,0 +1,64 @@
$ErrorActionPreference = 'Stop'
$root = [System.IO.Path]::GetFullPath((Split-Path (Split-Path $PSScriptRoot -Parent) -Parent))
try {
if (-not [Environment]::Is64BitOperatingSystem -or $env:PROCESSOR_ARCHITECTURE -eq 'ARM64') {
throw '此离线包需要 Intel/AMD 64 位 Windows 电脑。'
}
$manifestPath = Join-Path $root 'manifest.json'
if (-not (Test-Path -LiteralPath $manifestPath -PathType Leaf)) {
throw '未找到离线包清单 manifest.json。请解压完整离线包后再启动。'
}
$manifest = Get-Content -Raw -Encoding UTF8 $manifestPath | ConvertFrom-Json
$maxRelativePath = 0
foreach ($entry in $manifest.files) {
if ($null -eq $entry.path -or $entry.path -isnot [string] -or [string]::IsNullOrWhiteSpace($entry.path)) {
throw '离线包清单包含无效文件路径。请重新解压完整离线包。'
}
if ($entry.path.Length -gt $maxRelativePath) { $maxRelativePath = $entry.path.Length }
}
if ($maxRelativePath -eq 0) { throw '离线包清单没有文件路径。请重新解压完整离线包。' }
$maxRootLength = 248 - $maxRelativePath
if ($root.Length -gt $maxRootLength) {
$recommended = if ($env:LOCALAPPDATA) { Join-Path $env:LOCALAPPDATA 'QualityCenter' } else { '当前用户可写的较短本地 NTFS 目录' }
throw "部署目录路径过长:根路径和清单最长相对路径不能超过 248 个字符。请尝试 $recommended;若该路径仍过长,请选择更短的当前用户可写 NTFS 目录。"
}
$driveRoot = [System.IO.Path]::GetPathRoot($root)
$drive = [System.IO.DriveInfo]::new($driveRoot)
if ($drive.DriveType -ne [System.IO.DriveType]::Fixed -or $drive.DriveFormat -ne 'NTFS') {
throw '请将离线包解压到本机固定 NTFS 磁盘上的当前用户可写目录。'
}
$python = Join-Path $root 'runtimes\python\python.exe'
if (-not (Test-Path -LiteralPath $python -PathType Leaf)) {
throw '未找到随包提供的 Python 运行环境。请重新解压完整离线包。'
}
$env:PYTHONUTF8 = '1'
$env:PYTHONIOENCODING = 'utf-8'
$env:QUALITY_UPDATE_ENABLED = 'true'
# Recover an interrupted app-local update before reading mutable deployment
# configuration or starting the local service processes.
& $python (Join-Path $root 'deploy\bootstrap\update-recovery.py')
if ($LASTEXITCODE -ne 0) { throw '更新恢复仍未完成。请保留本地更新备份后重试。' }
$port = 5280
$config = Join-Path $root '.runtime\deployment.json'
if (Test-Path -LiteralPath $config -PathType Leaf) {
$port = [int](Get-Content -Raw -Encoding UTF8 $config | ConvertFrom-Json).webPort
}
if ($port -lt 1024 -or $port -gt 65535) { throw '网页端口配置无效。' }
if (Test-Path (Join-Path $root 'migration\manifest.json')) {
& $python (Join-Path $root 'deploy\offline\restore_migration.py')
if ($LASTEXITCODE -ne 0) { throw '数据恢复未完成。现有数据库已保留,请查看 runtimedata\logs。' }
}
Write-Host '正在以当前用户启动本机应用进程;不会安装系统组件、创建 Windows 服务或更改防火墙规则。'
Write-Host "网页服务使用内网监听,端口为 $port。其他设备能否访问取决于 Windows 当前防火墙和网络策略。"
& $python (Join-Path $root 'deploy\offline\control.py') start --lan
if ($LASTEXITCODE -ne 0) { throw '启动未完成。请查看 runtimedata\logs\supervisor.log。' }
} catch {
Write-Host $_ -ForegroundColor Red
exit 1
}