完善:同月多文件导入与当前用户离线迁移更新
This commit is contained in:
@@ -0,0 +1,36 @@
|
||||
# 质量中心离线部署
|
||||
|
||||
此目录是 Windows / Linux 原生离线包的打包与启动工具。编译后的前端、Spring Boot fat JAR、Java 21、MySQL 8.4、Python 3.11.15 和固定版本的 AI 依赖全部随包交付;目标机不执行 pip、pnpm、Maven 或联网下载。不需要 Docker。
|
||||
|
||||
Windows 推荐将离线包解压到 `%LOCALAPPDATA%\QualityCenter`,或本机固定 NTFS 磁盘上更短的当前用户可写目录。启动器会读取包内 `manifest.json`,要求部署根路径长度加最长相对文件路径不超过 248 个字符;若用户名较长导致 `%LOCALAPPDATA%\QualityCenter` 超限,请选择更短的用户可写目录。请用当前用户运行 `windows-start.ps1`;服务进程在当前用户会话中启动,运行环境随包提供,不安装系统组件、不注册全局服务,也不修改 Windows 防火墙。网页服务会监听内网网卡并显示本机及内网访问链接;MySQL 和后端接口只监听 `127.0.0.1`。其他设备能否访问网页取决于系统现有防火墙和网络策略,需要放行时由用户或管理员按组织策略处理。
|
||||
|
||||
Linux 默认只允许本机访问,使用 `./start.sh --lan` 开启内网监听;已有防火墙策略由部署者按实际网段配置。使用普通用户启动,不能以 root 运行 MySQL。
|
||||
|
||||
`build_bundle.py --work output/offline-20261104` 使用隔离构建目录中的 JAR / 前端产物及已下载的官方发行文件组装包。`downloads/assets.json` 记录来源及摘要,`requirements-*.lock` 固定各平台依赖,`wheels-*` 是已通过 pip 哈希验证的平台 wheel。该命令不会读取本机数据库、用户配置、API 密钥或 AI 会话。
|
||||
|
||||
为保留本机运行环境,跨平台适配只应用在包内的 Python 脚本副本上;原有 macOS 脚本不被改写。适配包括 Windows 的文件锁、进程创建时间校验和 NTFS 继承权限,以及随包 Python / nao-core 的相对位置。
|
||||
|
||||
包内 `manifest.json` 记录每个只读文件的 SHA-256 和符号链接目标。首次及后续启动都会检查文件完整性和平台原生程序;运行数据单独保存在 `.runtime/` 和 `runtimedata/`。初始管理员密码在目标机随机生成,不使用公共默认密码。
|
||||
|
||||
包内不包含业务数据库、工作簿、现有用户、模型密钥或本机私有配置。全离线时可使用常规查询、导入和分析功能;AI 调用在线模型时需要网络,或自行配置可达的内网模型服务。代码执行沙箱和开发测试工具不在业务部署范围内。
|
||||
|
||||
不能在服务运行时移动安装目录。迁移业务数据应使用逻辑备份与对应版本的恢复流程,不得直接将 macOS 的 MySQL 数据目录复制到其他系统。
|
||||
|
||||
## Windows 增量更新包(维护人员)
|
||||
|
||||
离线签名工具读取 Windows x64 部署包的 `manifest.json`,只打包 `update_manager.program_path` 认可的完整程序清单,并使用包内运行环境和 wheel 信息计算运行时指纹。私有配置、业务数据、迁移文件、启动引导和密钥不会进入更新包。普通发布的序号必须大于部署包当前的 `updateSequence`。首次通道种子可显式加 `--allow-baseline`,但版本和序号必须分别与 `updateVersion`、`updateSequence` 完全相同;低序号始终拒绝。私钥必须是与包内更新公钥匹配的 Ed25519 PEM 文件,并保存在部署包及产物目录之外。
|
||||
|
||||
```sh
|
||||
python3 deploy/offline/publish_update.py \
|
||||
--root /path/to/QualityCenter-Windows-x64 \
|
||||
--version 2026.11.04.2 \
|
||||
--sequence 2 \
|
||||
--private-key /secure/path/update-private-key.pem \
|
||||
--output-dir /path/to/new-update-output
|
||||
```
|
||||
|
||||
若需要为 `updateVersion=2026.11.04.1`、`updateSequence=1` 的原始包签署首个通道基线,使用与该包完全相同的版本和序号,并显式传入 `--allow-baseline`。不要修改包内 manifest 来绕过检查。
|
||||
|
||||
工具只在本地产生 `quality-update-<version>.zip` 和签名 `latest.json`,不会上传或发布文件。每个版本请使用新的输出目录;发布到更新服务器由维护人员另行完成。
|
||||
|
||||
参考:[Java 21 官方发行](https://adoptium.net/temurin/releases?version=21)、[MySQL Windows 安装要求](https://dev.mysql.com/doc/refman/8.4/en/windows-installation.html)、[MySQL Linux 二进制安装](https://dev.mysql.com/doc/refman/8.4/en/binary-installation.html)、[Python standalone](https://github.com/astral-sh/python-build-standalone)、[nao-core 0.3.26](https://pypi.org/project/nao-core/0.3.26/)、[Microsoft 运行库](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist/)。
|
||||
@@ -0,0 +1,330 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Assemble native offline packages from reviewed builds and verified downloads."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import stat
|
||||
import subprocess
|
||||
import tarfile
|
||||
import tempfile
|
||||
import zipfile
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
TEMPLATES = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def digest(path, algorithm='sha256'):
|
||||
with path.open('rb') as stream:
|
||||
return hashlib.file_digest(stream, algorithm).hexdigest()
|
||||
|
||||
|
||||
def extract(archive, target):
|
||||
target.mkdir(parents=True)
|
||||
with tempfile.TemporaryDirectory(dir=target.parent) as directory:
|
||||
stage = Path(directory)
|
||||
if archive.suffix == '.zip':
|
||||
with zipfile.ZipFile(archive) as source:
|
||||
source.extractall(stage)
|
||||
else:
|
||||
with tarfile.open(archive) as source:
|
||||
# Terminfo contains case-distinct names. They alias on a Mac's
|
||||
# default filesystem and are unused by this noninteractive service.
|
||||
members = [member for member in source.getmembers()
|
||||
if '/share/terminfo/' not in member.name and not member.name.endswith('/share/terminfo')]
|
||||
source.extractall(stage, members=members, filter='data')
|
||||
roots = list(stage.iterdir())
|
||||
if len(roots) != 1 or not roots[0].is_dir():
|
||||
raise RuntimeError('Expected one archive root: ' + archive.name)
|
||||
shutil.copytree(roots[0], target, symlinks=True, dirs_exist_ok=True)
|
||||
|
||||
|
||||
def extract_deb(archive, target):
|
||||
raw = archive.read_bytes()
|
||||
if raw[:8] != b'!<arch>\n':
|
||||
raise RuntimeError('Invalid Debian archive')
|
||||
position = 8
|
||||
while position + 60 <= len(raw):
|
||||
header = raw[position:position + 60]
|
||||
name = header[:16].decode().strip().rstrip('/')
|
||||
length = int(header[48:58].decode().strip())
|
||||
position += 60
|
||||
data = raw[position:position + length]
|
||||
position += length + length % 2
|
||||
if name.startswith('data.tar'):
|
||||
with tempfile.TemporaryDirectory(dir=target.parent) as directory:
|
||||
if name.endswith('.zst'):
|
||||
import zstandard
|
||||
with zstandard.ZstdDecompressor().stream_reader(io.BytesIO(data)) as reader:
|
||||
with tarfile.open(fileobj=reader, mode='r|') as source:
|
||||
source.extractall(directory, filter='data')
|
||||
else:
|
||||
with tarfile.open(fileobj=io.BytesIO(data)) as source:
|
||||
source.extractall(directory, filter='data')
|
||||
for path in Path(directory).rglob('*'):
|
||||
if '.so' in path.name and (path.is_file() or path.is_symlink()):
|
||||
dest = target / path.name
|
||||
if path.is_symlink():
|
||||
dest.symlink_to(Path(path.readlink()).name)
|
||||
else:
|
||||
shutil.copy2(path, dest)
|
||||
docs = Path(directory) / 'usr/share/doc'
|
||||
if docs.exists():
|
||||
shutil.copytree(docs, target / 'licenses', dirs_exist_ok=True, symlinks=True)
|
||||
return
|
||||
raise RuntimeError('No data member in ' + archive.name)
|
||||
|
||||
|
||||
def install_wheel(wheel, python_root, windows):
|
||||
site = python_root / ('Lib/site-packages' if windows else 'lib/python3.11/site-packages')
|
||||
site.mkdir(parents=True, exist_ok=True)
|
||||
with zipfile.ZipFile(wheel) as source:
|
||||
for member in source.infolist():
|
||||
if member.is_dir():
|
||||
continue
|
||||
relative = Path(member.filename)
|
||||
if relative.is_absolute() or '..' in relative.parts:
|
||||
raise RuntimeError('Wheel path escape')
|
||||
target = site / relative
|
||||
if relative.parts[0].endswith('.data'):
|
||||
scheme = relative.parts[1]
|
||||
prefix = {'purelib': site, 'platlib': site, 'data': python_root,
|
||||
'scripts': python_root / ('Scripts' if windows else 'bin'),
|
||||
'headers': python_root / 'include'}[scheme]
|
||||
target = prefix.joinpath(*relative.parts[2:])
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
with source.open(member) as reader, target.open('wb') as writer:
|
||||
shutil.copyfileobj(reader, writer)
|
||||
mode = member.external_attr >> 16
|
||||
target.chmod((mode & 0o777) or 0o644)
|
||||
|
||||
|
||||
def adapt_runtime(source):
|
||||
def replace(old, new, count=None):
|
||||
nonlocal source
|
||||
actual = source.count(old)
|
||||
if not actual or (count is not None and actual != count):
|
||||
raise RuntimeError('Runtime source changed; inspect the portability patch: ' + old)
|
||||
source = source.replace(old, new)
|
||||
|
||||
replace('import fcntl\n', 'import runtime_platform as _host\nfrom runtime_platform import fcntl\n', 1)
|
||||
replace('PYTHON = VENV / "bin" / "python"',
|
||||
'PYTHON = Path(os.environ.get("QUALITY_ASSISTANT_PYTHON", str(VENV / "bin" / "python")))', 1)
|
||||
old = 'VENV / "lib" / "python3.11" / "site-packages" / "nao_core"'
|
||||
replace(old, 'Path(os.environ.get("QUALITY_ASSISTANT_PACKAGE", str(' + old + ')))', 2)
|
||||
replace('/ "nao-chat-server"', '/ ("nao-chat-server.exe" if os.name == "nt" else "nao-chat-server")', 1)
|
||||
source, count = re.subn(r'\b(info|directory_info)\.st_mode & 0o077',
|
||||
lambda match: '_host.private_bits(' + match[1] + '.st_mode)', source)
|
||||
if count != 3:
|
||||
raise RuntimeError('Runtime private-permission checks changed; inspect before packaging')
|
||||
replace('os.fchmod(', '_host.fchmod(')
|
||||
replace('os.getpgid(pid) != pid', 'not _host.group_leader(pid)', 1)
|
||||
replace('os.killpg(', '_host.killpg(')
|
||||
replace('signal.SIGKILL', 'getattr(signal, "SIGKILL", 9)')
|
||||
replace('def _ps_value(pid: int, field: str) -> str | None:\n',
|
||||
'def _ps_value(pid: int, field: str) -> str | None:\n'
|
||||
' if os.name == "nt":\n return _host.ps_value(pid, field)\n', 1)
|
||||
compile(source, 'nao_quality_runtime.py', 'exec')
|
||||
return source
|
||||
|
||||
|
||||
def adapt_script(name, source):
|
||||
if name == 'nao_quality_runtime.py':
|
||||
return adapt_runtime(source)
|
||||
if name == 'analysis_visual_evidence.py':
|
||||
source = source.replace('import stat\n', 'import stat\nimport runtime_platform as _host\n', 1)
|
||||
source = source.replace('os.O_NOFOLLOW', 'getattr(os, "O_NOFOLLOW", 0)')
|
||||
source = source.replace('os.O_NONBLOCK', 'getattr(os, "O_NONBLOCK", 0)')
|
||||
source = source.replace('before.st_mode & 0o077', '_host.private_bits(before.st_mode)')
|
||||
source = source.replace('before.st_uid != os.getuid()', '_host.owner_mismatch(before)')
|
||||
source = source.replace(' fd = os.open(path,',
|
||||
' if path.is_symlink():\n raise ValueError("invalid evidence link")\n fd = os.open(path,', 1)
|
||||
elif name == 'nao_evidence_finalizer.py':
|
||||
source = source.replace('import copy\n', 'import copy\nimport runtime_platform as _host\n', 1)
|
||||
source = source.replace('path.stat().st_mode & 0o077', '_host.private_bits(path.stat().st_mode)')
|
||||
compile(source, name, 'exec')
|
||||
return source
|
||||
|
||||
|
||||
def windows_local_runtime(root):
|
||||
"""Use the signed vendor JRE's redistributable runtime; never install globally."""
|
||||
libraries = ('msvcp140.dll', 'vcruntime140.dll', 'vcruntime140_1.dll')
|
||||
for name in libraries:
|
||||
source = root / 'runtimes/java/bin' / name
|
||||
if not source.is_file():
|
||||
raise RuntimeError('The official JRE is missing an app-local Microsoft runtime: ' + name)
|
||||
for folder in ('runtimes/mysql/bin', 'runtimes/mysql/lib'):
|
||||
shutil.copy2(source, root / folder / name)
|
||||
mysql = root / 'runtimes/mysql'
|
||||
# Debug-only tools depend on non-redistributable Microsoft debug runtimes.
|
||||
for path in sorted(mysql.rglob('*'), reverse=True):
|
||||
if path.is_file() and (any(part.lower() == 'debug' for part in path.relative_to(mysql).parts)
|
||||
or '-debug.' in path.name.lower()):
|
||||
path.unlink()
|
||||
elif path.is_dir() and path.name.lower() == 'debug':
|
||||
shutil.rmtree(path)
|
||||
return {'name': 'app-local-msvc', 'os': 'windows', 'source': 'bundled-official-temurin-jre',
|
||||
'libraries': {name: digest(root / 'runtimes/java/bin' / name) for name in libraries}}
|
||||
|
||||
|
||||
def payload(root, work, osname, assets):
|
||||
windows = osname == 'windows'
|
||||
app = root / 'app'
|
||||
app.mkdir(parents=True)
|
||||
jars = list((work / 'source/backend/target').glob('*.jar'))
|
||||
if len(jars) != 1:
|
||||
raise RuntimeError('Expected one freshly built backend JAR')
|
||||
with zipfile.ZipFile(jars[0]) as archive:
|
||||
if 'BOOT-INF/classes/verification/original-workbook-acceptance.json' in archive.namelist():
|
||||
raise RuntimeError('Exclude the private workbook acceptance input before building a distributable JAR')
|
||||
shutil.copy2(jars[0], app / 'backend.jar')
|
||||
shutil.copytree(work / 'source/web/dist', app / 'web')
|
||||
for name in ('java', 'mysql', 'python'):
|
||||
kind = 'jre' if name == 'java' else name
|
||||
item = next(a for a in assets if a['name'] == kind and a['os'] == osname)
|
||||
archive = work / 'downloads' / item['file']
|
||||
if digest(archive) != item['sha256']:
|
||||
raise RuntimeError('Download checksum mismatch: ' + item['file'])
|
||||
extract(archive, root / 'runtimes' / name)
|
||||
wheels = sorted((work / ('wheels-' + osname)).glob('*.whl'))
|
||||
if len(wheels) < 60:
|
||||
raise RuntimeError('The offline Python dependency set is incomplete')
|
||||
lock_text = (work / f'requirements-{osname}.lock').read_text()
|
||||
for wheel in wheels:
|
||||
if '--hash=sha256:' + digest(wheel) not in lock_text:
|
||||
raise RuntimeError('Wheel absent from hash lock: ' + wheel.name)
|
||||
install_wheel(wheel, root / 'runtimes/python', windows)
|
||||
scripts = root / 'scripts'
|
||||
scripts.mkdir()
|
||||
for pattern in ('nao_*.py', 'analysis_visual*.py'):
|
||||
for path in (ROOT / 'scripts').glob(pattern):
|
||||
(scripts / path.name).write_text(adapt_script(path.name, path.read_text()), encoding='utf-8')
|
||||
for name in ('dbgpt_trial_data.py', 'local_api.py', 'project.py'):
|
||||
shutil.copy2(ROOT / 'scripts' / name, scripts / name)
|
||||
shutil.copy2(TEMPLATES / 'runtime_platform.py', scripts / 'runtime_platform.py')
|
||||
policy = root / 'backend/src/main/resources/analysis-runtime-policy.json'
|
||||
policy.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(ROOT / 'backend/src/main/resources/analysis-runtime-policy.json', policy)
|
||||
shutil.copytree(ROOT / 'ai/nao-quality', root / 'ai/nao-quality', ignore=shutil.ignore_patterns('.DS_Store'))
|
||||
deploy = root / 'deploy/offline'
|
||||
deploy.mkdir(parents=True)
|
||||
for name in ('control.py', 'gateway.py', 'runtime_platform.py', 'migration_common.py',
|
||||
'restore_migration.py', 'update_manager.py', 'update-public-key.pem',
|
||||
'windows-start.ps1', 'windows-prepare.ps1', 'README.md'):
|
||||
if not windows and name.endswith('.ps1'):
|
||||
continue
|
||||
source = TEMPLATES / name
|
||||
if name.endswith('.ps1'):
|
||||
(deploy / name).write_text(source.read_text(), encoding='utf-8-sig')
|
||||
else:
|
||||
shutil.copy2(source, deploy / name)
|
||||
shutil.copy2(work / f'requirements-{osname}.lock', root / 'requirements.lock')
|
||||
if windows:
|
||||
local_runtime = windows_local_runtime(root)
|
||||
shutil.copy2(root / 'runtimes/python/python.exe', root / 'runtimes/python/python3.exe')
|
||||
bootstrap = root / 'deploy/bootstrap'
|
||||
bootstrap.mkdir()
|
||||
shutil.copy2(TEMPLATES / 'update_bootstrap.py', bootstrap / 'update-recovery.py')
|
||||
for name in ('windows-start.ps1', 'windows-prepare.ps1'):
|
||||
(bootstrap / name).write_text((TEMPLATES / name).read_text(), encoding='utf-8-sig')
|
||||
(root / '一键部署启动.cmd').write_bytes(
|
||||
b'@echo off\r\nchcp 65001 >nul\r\npowershell.exe -NoProfile -ExecutionPolicy Bypass -File "%~dp0deploy\\bootstrap\\windows-start.ps1"\r\n'
|
||||
b'if errorlevel 1 (echo Startup failed. See the message above.)\r\npause\r\n')
|
||||
for name, action in (('停止服务.cmd', 'stop'), ('查看状态.cmd', 'status'), ('检查环境.cmd', 'verify')):
|
||||
(root / name).write_bytes(('@echo off\r\nchcp 65001 >nul\r\nset PYTHONUTF8=1\r\nset PYTHONIOENCODING=utf-8\r\n'
|
||||
'"%~dp0runtimes\\python\\python.exe" "%~dp0deploy\\offline\\control.py" ' + action + '\r\npause\r\n').encode())
|
||||
else:
|
||||
libs = root / 'runtimes/linux-libs'
|
||||
libs.mkdir()
|
||||
for item in assets:
|
||||
if item.get('kind') == 'deb':
|
||||
archive = work / 'downloads' / item['file']
|
||||
if digest(archive) != item['sha256']:
|
||||
raise RuntimeError('Debian library checksum mismatch')
|
||||
extract_deb(archive, libs)
|
||||
for name, action in (('start.sh', 'start'), ('stop.sh', 'stop'), ('status.sh', 'status'), ('verify.sh', 'verify')):
|
||||
(root / name).write_text('#!/bin/sh\nset -eu\nROOT=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)\n'
|
||||
'if [ "$(id -u)" -eq 0 ]; then echo "Run as a normal user, not root." >&2; exit 1; fi\n'
|
||||
'export PYTHONUTF8=1 PYTHONIOENCODING=utf-8 PYTHONDONTWRITEBYTECODE=1\n'
|
||||
'export LD_LIBRARY_PATH="$ROOT/runtimes/linux-libs:$ROOT/runtimes/mysql/lib:$ROOT/runtimes/mysql/lib/private"\n'
|
||||
'exec "$ROOT/runtimes/python/bin/python3" "$ROOT/deploy/offline/control.py" ' + action + ' "$@"\n')
|
||||
(root / name).chmod(0o755)
|
||||
system = ('Windows 10/11 或 Windows Server 2019/2022/2025,Intel/AMD x64。仅当前用户运行,无需管理员权限。'
|
||||
if windows else 'Ubuntu 22.04/24.04、Debian 12/13 的 Intel/AMD x86_64。需要 glibc 2.35 及以上;不支持 Alpine/musl、ARM 或旧版 CentOS。')
|
||||
usage = ('解压到当前用户的 %LOCALAPPDATA%\\QualityCenter 等较短的本地 NTFS 路径,双击“一键部署启动.cmd”。不要在压缩包中直接打开。\n'
|
||||
'运行库随包提供;不安装系统服务,不修改系统环境变量、注册表或防火墙,不申请管理员权限。\n'
|
||||
'启动后自动打开浏览器,命令窗口显示本机和内网地址。关闭此窗口不会停止服务。\n'
|
||||
'其他电脑连接同一内网后,用浏览器打开显示的 http://内网IP:5280 即可。\n'
|
||||
'停止服务请双击“停止服务.cmd”;重新启动仍双击原入口。\n' if windows else
|
||||
'解压后以普通用户运行 ./start.sh;内网共享运行 ./start.sh --lan。\n'
|
||||
'同事用浏览器打开显示的 http://内网IP:5280;如系统有防火墙,仅对可信内网放行该端口。\n'
|
||||
'停止运行 ./stop.sh,查看状态运行 ./status.sh,检查完整性运行 ./verify.sh。\n')
|
||||
(root / '使用说明.txt').write_text('质量中心离线部署包\n\n系统要求\n' + system + '\n建议内存 8GB 以上,至少预留 5GB 磁盘及后续业务数据空间。\n\n'
|
||||
+ usage + '\n登录信息保存在 runtimedata/登录信息.txt,初始管理员密码在本机随机生成。登录后请修改密码。\n'
|
||||
'首次数据库为空,不包含原电脑的用户、业务数据、来源文件和模型密钥。通过网页上传授权的数据文件。\n'
|
||||
'普通功能和安装不需要外网;AI 在线模型调用需要网络,或自行配置内网模型。\n'
|
||||
'不要在服务运行时移动目录,不要覆盖 .runtime 和 runtimedata。数据迁移使用逻辑备份,不能复制 macOS MySQL 原始数据目录。\n'
|
||||
'内网访问需要主机保持开机、服务运行、网络互通;本机 MySQL 和内部 API 端口不对内网开放。\n'
|
||||
'默认网页端口 5280。更改 .runtime/deployment.json 中的 webPort 后重新运行入口。内网连接受现有防火墙规则限制,程序不会自动更改这些规则。\n'
|
||||
'详细技术说明见 deploy/offline/README.md。\n\n'
|
||||
'验证范围:前后端隔离构建及本机自动检查已完成;Windows/Linux 原生首次启动和跨电脑内网访问需要在目标机实测。\n', encoding='utf-8-sig')
|
||||
metadata = {'platform': osname, 'architecture': 'x86_64', 'builtAtUtc': datetime.now(timezone.utc).isoformat(),
|
||||
'updateVersion': '2026.11.04.1', 'updateSequence': 1, 'installationScope': 'current-user', 'requiresAdministrator': False,
|
||||
'databaseIncluded': False, 'credentialsIncluded': False, 'requiresDownloadAtInstall': False,
|
||||
'nativeTargetSmokeTest': 'not-run-on-macos-host',
|
||||
'runtimes': [{k: v for k, v in a.items() if k not in ('kind',)} for a in assets if a['os'] == osname and a['name'] != 'vcredist'] + ([local_runtime] if windows else []),
|
||||
'pythonWheels': [{'file': p.name, 'sha256': digest(p)} for p in wheels], 'files': []}
|
||||
for path in sorted(root.rglob('*')):
|
||||
if path.is_symlink():
|
||||
metadata['files'].append({'path': path.relative_to(root).as_posix(), 'symlink': str(path.readlink())})
|
||||
elif path.is_file():
|
||||
metadata['files'].append({'path': path.relative_to(root).as_posix(), 'sha256': digest(path)})
|
||||
(root / 'manifest.json').write_text(json.dumps(metadata, ensure_ascii=False, indent=2) + '\n', encoding='utf-8')
|
||||
return metadata
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--work', type=Path, required=True)
|
||||
parser.add_argument('--platform', choices=('windows', 'linux', 'both'), default='both')
|
||||
parser.add_argument('--assemble-only', action='store_true')
|
||||
args = parser.parse_args()
|
||||
work = args.work.resolve()
|
||||
assets = json.loads((work / 'downloads/assets.json').read_text())
|
||||
platforms = ('windows', 'linux') if args.platform == 'both' else (args.platform,)
|
||||
summaries = []
|
||||
for osname in platforms:
|
||||
name = 'QualityCenter-' + ('Windows' if osname == 'windows' else 'Linux') + '-x64'
|
||||
directory = work / name
|
||||
if directory.exists():
|
||||
raise RuntimeError('Preserving an existing bundle; choose an empty work directory: ' + str(directory))
|
||||
print('Assembling ' + name, flush=True)
|
||||
metadata = payload(directory, work, osname, assets)
|
||||
print('Files verified: ' + str(len(metadata['files'])), flush=True)
|
||||
if args.assemble_only:
|
||||
continue
|
||||
if osname == 'windows':
|
||||
archive = work / (name + '.zip')
|
||||
with zipfile.ZipFile(archive, 'w', zipfile.ZIP_DEFLATED, compresslevel=6) as target:
|
||||
for path in sorted(directory.rglob('*')):
|
||||
if path.is_file():
|
||||
target.write(path, path.relative_to(work).as_posix())
|
||||
else:
|
||||
archive = work / (name + '.tar.gz')
|
||||
with tarfile.open(archive, 'w:gz', compresslevel=6) as target:
|
||||
target.add(directory, arcname=name)
|
||||
summaries.append({'file': archive.name, 'bytes': archive.stat().st_size, 'sha256': digest(archive)})
|
||||
print(json.dumps(summaries[-1]), flush=True)
|
||||
if summaries:
|
||||
(work / 'SHA256SUMS.txt').write_text(''.join(a['sha256'] + ' ' + a['file'] + '\n' for a in summaries))
|
||||
(work / 'packages.json').write_text(json.dumps(summaries, indent=2) + '\n')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,486 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Offline, unprivileged service supervisor for the standalone distribution."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import contextlib
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import re
|
||||
import secrets
|
||||
import signal
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from urllib.request import ProxyHandler, build_opener
|
||||
|
||||
import psutil
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / 'scripts'))
|
||||
import runtime_platform as host
|
||||
from gateway import make_server
|
||||
|
||||
PRIVATE = ROOT / '.runtime'
|
||||
DATA = ROOT / 'runtimedata'
|
||||
CONFIG = PRIVATE / 'deployment.json'
|
||||
STATE = PRIVATE / 'service.json'
|
||||
STOP = PRIVATE / 'stop-request.json'
|
||||
IS_WINDOWS = os.name == 'nt'
|
||||
EXE = '.exe' if IS_WINDOWS else ''
|
||||
PYTHON = ROOT / 'runtimes/python' / ('python.exe' if IS_WINDOWS else 'bin/python3')
|
||||
JAVA = ROOT / 'runtimes/java/bin' / ('java' + EXE)
|
||||
MYSQL = ROOT / 'runtimes/mysql/bin' / ('mysql' + EXE)
|
||||
MYSQLD = ROOT / 'runtimes/mysql/bin' / ('mysqld' + EXE)
|
||||
ADMIN = ROOT / 'runtimes/mysql/bin' / ('mysqladmin' + EXE)
|
||||
OPENER = build_opener(ProxyHandler({}))
|
||||
|
||||
|
||||
def atomic(path, value):
|
||||
temporary = path.with_name(path.name + '.' + secrets.token_hex(5) + '.tmp')
|
||||
fd = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
with os.fdopen(fd, 'w', encoding='utf-8') as stream:
|
||||
json.dump(value, stream, ensure_ascii=False, indent=2)
|
||||
temporary.replace(path)
|
||||
|
||||
|
||||
def private_directory(path):
|
||||
if path.is_symlink():
|
||||
raise RuntimeError('运行目录不能是符号链接。')
|
||||
path.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
if IS_WINDOWS:
|
||||
result = subprocess.run(['whoami', '/user', '/fo', 'csv', '/nh'], capture_output=True, check=True)
|
||||
sid = re.search(rb'S-1-\d+(?:-\d+)+', result.stdout)
|
||||
if not sid:
|
||||
raise RuntimeError('无法确定当前 Windows 用户。')
|
||||
subprocess.run(['icacls', str(path), '/inheritance:r', '/grant:r',
|
||||
'*' + sid.group().decode() + ':(OI)(CI)F', '*S-1-5-18:(OI)(CI)F', '/Q'],
|
||||
stdout=subprocess.DEVNULL, check=True)
|
||||
else:
|
||||
path.chmod(0o700)
|
||||
|
||||
|
||||
def configuration():
|
||||
if CONFIG.exists():
|
||||
value = json.loads(CONFIG.read_text(encoding='utf-8'))
|
||||
else:
|
||||
value = {'mysqlPort': 53306, 'backendPort': 5281, 'webPort': 5280,
|
||||
'webBind': '0.0.0.0' if IS_WINDOWS else '127.0.0.1',
|
||||
'mysqlPassword': secrets.token_urlsafe(32), 'rootPassword': secrets.token_urlsafe(32),
|
||||
'adminUsername': 'admin', 'adminPassword': secrets.token_urlsafe(24)}
|
||||
atomic(CONFIG, value)
|
||||
value.setdefault('mysqlDatabase', 'quality_platform')
|
||||
if not re.fullmatch(r'quality_(?:platform|restore_[a-f0-9]{12,32})', value['mysqlDatabase']):
|
||||
raise RuntimeError('部署数据库名称无效。')
|
||||
relative_archive = Path(value.get('archiveDirectory', 'runtimedata/archive'))
|
||||
if relative_archive.is_absolute() or '..' in relative_archive.parts or not (ROOT / relative_archive).resolve().is_relative_to(DATA.resolve()):
|
||||
raise RuntimeError('来源归档目录无效。')
|
||||
for key in ('mysqlPort', 'backendPort', 'webPort'):
|
||||
if type(value.get(key)) is not int or not 1024 <= value[key] <= 65535:
|
||||
raise RuntimeError('部署端口配置无效。')
|
||||
if value['backendPort'] not in (5281, 5283, 5284, 5285):
|
||||
raise RuntimeError('接口端口必须为 5281、5283、5284 或 5285。')
|
||||
if len({value[k] for k in ('mysqlPort', 'backendPort', 'webPort')}) != 3:
|
||||
raise RuntimeError('数据库、接口和网页端口不能重复。')
|
||||
if value.get('webBind') not in ('127.0.0.1', '0.0.0.0'):
|
||||
raise RuntimeError('网页监听配置无效。')
|
||||
for key in ('mysqlPassword', 'rootPassword', 'adminPassword'):
|
||||
if not re.fullmatch(r'[A-Za-z0-9_-]{20,100}', value.get(key, '')):
|
||||
raise RuntimeError('本机随机凭据配置无效,原配置已保留。')
|
||||
return value
|
||||
|
||||
|
||||
def environment(value):
|
||||
env = os.environ.copy()
|
||||
package = ROOT / 'runtimes/python' / ('Lib/site-packages' if IS_WINDOWS else 'lib/python3.11/site-packages') / 'nao_core'
|
||||
env.update({
|
||||
'JAVA_HOME': str(JAVA.parent.parent),
|
||||
'PATH': os.pathsep.join([str(PYTHON.parent), str(JAVA.parent), str(MYSQL.parent), env.get('PATH', '')]),
|
||||
'PYTHONUTF8': '1', 'PYTHONIOENCODING': 'utf-8', 'PYTHONDONTWRITEBYTECODE': '1',
|
||||
'QUALITY_PROJECT_ROOT': str(ROOT), 'QUALITY_ASSISTANT_PYTHON': str(PYTHON),
|
||||
'QUALITY_ASSISTANT_PACKAGE': str(package),
|
||||
'MYSQL_URL': f'jdbc:mysql://127.0.0.1:{value["mysqlPort"]}/{value.get("mysqlDatabase", "quality_platform")}?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC&forceConnectionTimeZoneToSession=true&useSSL=false&allowPublicKeyRetrieval=true',
|
||||
'MYSQL_USER': 'quality_app', 'MYSQL_PASSWORD': value['mysqlPassword'],
|
||||
'QUALITY_BOOTSTRAP_ADMIN_USERNAME': '' if value.get('restoredFromMigration') else value['adminUsername'],
|
||||
'QUALITY_BOOTSTRAP_ADMIN_PASSWORD': '' if value.get('restoredFromMigration') else value['adminPassword'],
|
||||
'QUALITY_ARCHIVE_ROOT': str(ROOT / value.get('archiveDirectory', 'runtimedata/archive')), 'QUALITY_ALLOWED_ROOTS': str(DATA / 'sources'),
|
||||
'QUALITY_ANALYSIS_RUNTIME_QUERY_PORT': str(value['backendPort']),
|
||||
'PORT': str(value['backendPort']), 'SERVER_ADDRESS': '127.0.0.1',
|
||||
'QUALITY_DATA_WORKFLOW_TEST_MODE': 'false',
|
||||
'MYSQL_LEGACY_TIMESTAMP_OFFSET_SECONDS': 'UNCONFIRMED',
|
||||
})
|
||||
env['QUALITY_DATA_WORKFLOW_TEST_MODE'] = str(value.get('dataWorkflowTestMode', False)).lower()
|
||||
env['QUALITY_ANALYSIS_AUTONOMOUS_ENABLED'] = str(value.get('analysisAutonomousEnabled', False)).lower()
|
||||
env['QUALITY_ANALYSIS_PROTOCOL_V1_ENABLED'] = str(value.get('analysisProtocolV1Enabled', False)).lower()
|
||||
env['QUALITY_UPDATE_ENABLED'] = 'true' if IS_WINDOWS else 'false'
|
||||
# Do not accidentally inherit a development or source-machine AI command.
|
||||
env.pop('QUALITY_ANALYSIS_NAO_COMMAND', None)
|
||||
env.pop('QUALITY_ANALYSIS_NAO_SCRIPT', None)
|
||||
env.pop('PYTHONHOME', None)
|
||||
env.pop('PYTHONPATH', None)
|
||||
if not IS_WINDOWS:
|
||||
env['LD_LIBRARY_PATH'] = os.pathsep.join([str(ROOT / 'runtimes/linux-libs'),
|
||||
str(ROOT / 'runtimes/mysql/lib'),
|
||||
str(ROOT / 'runtimes/mysql/lib/private')])
|
||||
return env
|
||||
|
||||
|
||||
def record(process, command):
|
||||
observed = psutil.Process(process.pid)
|
||||
return {'pid': process.pid, 'created': observed.create_time(),
|
||||
'executable': str(Path(observed.exe()).resolve()), 'args': [str(a) for a in command[1:]]}
|
||||
|
||||
|
||||
def matches(info):
|
||||
try:
|
||||
process = psutil.Process(int(info['pid']))
|
||||
return (process.is_running() and process.status() != psutil.STATUS_ZOMBIE
|
||||
and abs(process.create_time() - info['created']) < .01
|
||||
and os.path.normcase(str(Path(process.exe()).resolve())) == os.path.normcase(info['executable'])
|
||||
and process.cmdline()[1:] == info['args'])
|
||||
except (psutil.Error, KeyError, ValueError, TypeError):
|
||||
return False
|
||||
|
||||
|
||||
def read_state():
|
||||
try:
|
||||
return json.loads(STATE.read_text(encoding='utf-8'))
|
||||
except FileNotFoundError:
|
||||
return {}
|
||||
|
||||
|
||||
def free_port(port, bind='127.0.0.1'):
|
||||
# A recently closed TCP connection may remain in TIME_WAIT. Reject a live
|
||||
# listener, but allow a clean restart without waiting for that state to expire.
|
||||
try:
|
||||
connection = socket.create_connection(('127.0.0.1', port), timeout=.3)
|
||||
except OSError:
|
||||
pass
|
||||
else:
|
||||
connection.close()
|
||||
raise RuntimeError(f'端口 {port} 已被其他服务使用。')
|
||||
with socket.socket() as test:
|
||||
test.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
test.bind((bind, port))
|
||||
|
||||
|
||||
def mysql_command(value, binary=MYSQL):
|
||||
return [str(binary), '--no-defaults', '--protocol=TCP', '--host=127.0.0.1',
|
||||
'--port=' + str(value['mysqlPort']), '--user=root', '--connect-timeout=3']
|
||||
|
||||
|
||||
def sql(value, query):
|
||||
env = environment(value)
|
||||
env['MYSQL_PWD'] = value['rootPassword']
|
||||
return subprocess.run(mysql_command(value) + ['--batch', '--skip-column-names'], input=query,
|
||||
text=True, encoding='utf-8', stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
env=env, timeout=10)
|
||||
|
||||
|
||||
def verify(check_hashes=True):
|
||||
manifest = json.loads((ROOT / 'manifest.json').read_text(encoding='utf-8'))
|
||||
expected_os = 'windows' if IS_WINDOWS else platform.system().lower()
|
||||
if manifest['platform'] != expected_os or platform.machine().lower() not in ('amd64', 'x86_64'):
|
||||
raise RuntimeError('此包仅用于 ' + manifest['platform'] + ' 的 Intel/AMD 64 位电脑。')
|
||||
if check_hashes:
|
||||
for entry in manifest['files']:
|
||||
path = ROOT / entry['path']
|
||||
if entry.get('symlink') is not None:
|
||||
valid = path.is_symlink() and os.readlink(path) == entry['symlink']
|
||||
else:
|
||||
valid = path.is_file() and hashlib.file_digest(path.open('rb'), 'sha256').hexdigest() == entry['sha256']
|
||||
if not valid:
|
||||
raise RuntimeError('文件缺失或校验失败:' + entry['path'])
|
||||
value = configuration()
|
||||
env = environment(value)
|
||||
for command in ([JAVA, '-version'], [MYSQLD, '--no-defaults', '--version']):
|
||||
result = subprocess.run([str(a) for a in command], capture_output=True, env=env, timeout=30)
|
||||
if result.returncode:
|
||||
raise RuntimeError('运行环境检查失败:' + Path(command[0]).name + '。请查看安装说明中的系统要求。')
|
||||
probe = subprocess.run([str(PYTHON), '-c', 'import duckdb,yaml,fastapi,psutil,nao_core;import sys;assert sys.version_info[:3]==(3,11,15)'],
|
||||
capture_output=True, env=env, timeout=45)
|
||||
if probe.returncode:
|
||||
raise RuntimeError('AI Python 运行环境检查失败。')
|
||||
application_probe = subprocess.run([str(PYTHON), '-c',
|
||||
'import sys;sys.path.insert(0,sys.argv[1]);import nao_quality_runtime as r;'
|
||||
'import nao_native_run,nao_quality_runtime_mcp,nao_visual_recovery;'
|
||||
'assert len(r._config_fingerprint("https://offline.invalid/v1","offline-check","offline-check"))==64',
|
||||
str(ROOT / 'scripts')], capture_output=True, env=env, timeout=45)
|
||||
if application_probe.returncode:
|
||||
raise RuntimeError('AI 应用脚本或策略资源检查失败。')
|
||||
package = Path(env['QUALITY_ASSISTANT_PACKAGE'])
|
||||
native = subprocess.run([str(package / 'bin' / ('nao-chat-server' + EXE)), '--help'],
|
||||
capture_output=True, env=env, timeout=30)
|
||||
if native.returncode:
|
||||
raise RuntimeError('AI 原生服务无法运行,请检查系统版本和 CPU 支持。')
|
||||
return value
|
||||
|
||||
|
||||
def addresses(value):
|
||||
ips = set()
|
||||
if value['webBind'] == '0.0.0.0':
|
||||
for entries in psutil.net_if_addrs().values():
|
||||
for entry in entries:
|
||||
if entry.family == socket.AF_INET and not entry.address.startswith(('127.', '169.254.')):
|
||||
ips.add(entry.address)
|
||||
return ['http://127.0.0.1:' + str(value['webPort'])] + [f'http://{ip}:{value["webPort"]}' for ip in sorted(ips)]
|
||||
|
||||
|
||||
def write_login(value):
|
||||
path = DATA / '登录信息.txt'
|
||||
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, 'w', encoding='utf-8') as stream:
|
||||
if value.get('restoredFromMigration'):
|
||||
stream.write('质量中心\n' + '\n'.join(addresses(value)) + '\n业务数据已迁移,请使用原有账号和密码登录。\n')
|
||||
else:
|
||||
stream.write('质量中心\n' + '\n'.join(addresses(value)) + '\n用户名:' + value['adminUsername']
|
||||
+ '\n初始密码:' + value['adminPassword'] + '\n登录后请在个人账户修改密码。此文件仅记录初始密码,请勿分享。\n')
|
||||
|
||||
|
||||
def serve(database_only=False):
|
||||
lock = os.open(PRIVATE / 'supervisor.lock', os.O_RDWR | os.O_CREAT, 0o600)
|
||||
try:
|
||||
host.fcntl.flock(lock, host.fcntl.LOCK_EX | host.fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
raise RuntimeError('此目录的服务已经启动。') from None
|
||||
children = []
|
||||
value = configuration()
|
||||
state = {'supervisor': record(psutil.Process(), [sys.executable, *sys.argv]),
|
||||
'token': secrets.token_hex(24), 'ready': False, 'databaseOnly': database_only, 'children': children}
|
||||
stopped = threading.Event()
|
||||
for sig in (signal.SIGINT, signal.SIGTERM):
|
||||
signal.signal(sig, lambda *_: stopped.set())
|
||||
|
||||
def stopping():
|
||||
if STOP.exists():
|
||||
try:
|
||||
if json.loads(STOP.read_text(encoding='utf-8')).get('token') == state['token']:
|
||||
stopped.set()
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
return stopped.is_set()
|
||||
|
||||
def spawn(name, command):
|
||||
with (DATA / 'logs' / (name + '.log')).open('ab') as log:
|
||||
child = subprocess.Popen([str(a) for a in command], cwd=ROOT, env=environment(value),
|
||||
stdin=subprocess.DEVNULL, stdout=log, stderr=log)
|
||||
info = record(child, command)
|
||||
info['name'] = name
|
||||
children.append(info)
|
||||
atomic(STATE, state)
|
||||
return child
|
||||
|
||||
def wait_ready(child, check, seconds):
|
||||
end = time.monotonic() + seconds
|
||||
while time.monotonic() < end:
|
||||
if stopping():
|
||||
raise RuntimeError('启动已停止。')
|
||||
if child.poll() is not None:
|
||||
raise RuntimeError('服务启动失败,请查看 runtimedata/logs。')
|
||||
try:
|
||||
if check():
|
||||
return
|
||||
except (OSError, ValueError, subprocess.TimeoutExpired):
|
||||
pass
|
||||
time.sleep(.5)
|
||||
raise RuntimeError('服务启动超时,请查看 runtimedata/logs。')
|
||||
|
||||
try:
|
||||
for key in (('mysqlPort',) if database_only else ('mysqlPort', 'backendPort', 'webPort')):
|
||||
free_port(value[key], value['webBind'] if key == 'webPort' else '127.0.0.1')
|
||||
atomic(STATE, state)
|
||||
db = DATA / 'mysql'
|
||||
if not (db / 'auto.cnf').exists():
|
||||
if db.exists() and any(db.iterdir()):
|
||||
raise RuntimeError('数据库目录含有未完成的初始化文件,已保留原目录;请查看初始化日志。')
|
||||
db.mkdir(exist_ok=True)
|
||||
with (DATA / 'logs/mysql-initialize.log').open('ab') as log:
|
||||
result = subprocess.run([str(MYSQLD), '--no-defaults', '--initialize-insecure',
|
||||
'--basedir=' + str(MYSQLD.parent.parent), '--datadir=' + str(db),
|
||||
'--lower-case-table-names=1'], env=environment(value),
|
||||
stdout=log, stderr=log, timeout=180)
|
||||
if result.returncode:
|
||||
raise RuntimeError('数据库初始化失败,请查看 runtimedata/logs/mysql-initialize.log。')
|
||||
bootstrap = PRIVATE / 'bootstrap.sql'
|
||||
fd = os.open(bootstrap, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, 'w', encoding='utf-8') as stream:
|
||||
stream.write(f"ALTER USER 'root'@'localhost' IDENTIFIED BY '{value['rootPassword']}';\n"
|
||||
f"CREATE USER IF NOT EXISTS 'root'@'127.0.0.1' IDENTIFIED BY '{value['rootPassword']}';\n"
|
||||
"GRANT ALL PRIVILEGES ON *.* TO 'root'@'127.0.0.1' WITH GRANT OPTION;\n"
|
||||
f"CREATE DATABASE IF NOT EXISTS `{value['mysqlDatabase']}` CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;\n"
|
||||
f"CREATE USER IF NOT EXISTS 'quality_app'@'127.0.0.1' IDENTIFIED BY '{value['mysqlPassword']}';\n"
|
||||
f"GRANT ALL PRIVILEGES ON `{value['mysqlDatabase']}`.* TO 'quality_app'@'127.0.0.1';\n")
|
||||
mysql_args = [MYSQLD, '--no-defaults', '--basedir=' + str(MYSQLD.parent.parent), '--datadir=' + str(db),
|
||||
'--port=' + str(value['mysqlPort']), '--bind-address=127.0.0.1', '--mysqlx=OFF',
|
||||
'--lower-case-table-names=1', '--skip-name-resolve', '--skip-log-bin',
|
||||
'--max-connections=60', '--innodb-buffer-pool-size=' + ('1024M' if database_only else '256M'), '--init-file=' + bootstrap.as_posix()]
|
||||
if database_only:
|
||||
# The restored database is unpublished until validation and a clean
|
||||
# MySQL shutdown finish. Normal startup restores durable defaults.
|
||||
mysql_args += ['--innodb-redo-log-capacity=1G', '--innodb-flush-log-at-trx-commit=2']
|
||||
if not IS_WINDOWS:
|
||||
mysql_args.append('--socket=' + str(PRIVATE / 'mysql.sock'))
|
||||
database = spawn('mysql', mysql_args)
|
||||
wait_ready(database, lambda: sql(value, 'SELECT 1;').returncode == 0, 120)
|
||||
# The server does not reread this file after startup.
|
||||
bootstrap.unlink(missing_ok=True)
|
||||
if database_only:
|
||||
state['ready'] = True
|
||||
atomic(STATE, state)
|
||||
while not stopping():
|
||||
if database.poll() is not None:
|
||||
raise RuntimeError('恢复数据库服务已退出,请查看日志。')
|
||||
time.sleep(.25)
|
||||
return
|
||||
backend = spawn('backend', [JAVA, '-Djava.awt.headless=true', '-Xms128m', '-Xmx2048m', '-jar', ROOT / 'app/backend.jar'])
|
||||
|
||||
def backend_ready():
|
||||
with OPENER.open(f'http://127.0.0.1:{value["backendPort"]}/actuator/health', timeout=3) as response:
|
||||
return json.load(response).get('status') == 'UP'
|
||||
|
||||
wait_ready(backend, backend_ready, 900)
|
||||
server = make_server(ROOT / 'app/web', value['webBind'], value['webPort'], value['backendPort'])
|
||||
server.timeout = .5
|
||||
state['ready'] = True
|
||||
write_login(value)
|
||||
atomic(STATE, state)
|
||||
print('质量中心已启动:' + ' '.join(addresses(value)), flush=True)
|
||||
try:
|
||||
while not stopping():
|
||||
if database.poll() is not None or backend.poll() is not None:
|
||||
raise RuntimeError('后台服务已退出,请查看 runtimedata/logs。')
|
||||
server.handle_request()
|
||||
finally:
|
||||
server.server_close()
|
||||
finally:
|
||||
state['ready'] = False
|
||||
atomic(STATE, state)
|
||||
# Stop backend first, then its isolated AI processes, then the database.
|
||||
for info in reversed(children):
|
||||
if info['name'] != 'backend' or not matches(info):
|
||||
continue
|
||||
process = psutil.Process(info['pid'])
|
||||
descendants = process.children(recursive=True)
|
||||
process.terminate()
|
||||
try:
|
||||
process.wait(35)
|
||||
except psutil.TimeoutExpired:
|
||||
if matches(info):
|
||||
process.kill()
|
||||
for child in descendants:
|
||||
with contextlib.suppress(psutil.NoSuchProcess):
|
||||
child.terminate()
|
||||
with contextlib.suppress(OSError, subprocess.SubprocessError):
|
||||
subprocess.run([str(PYTHON), str(ROOT / 'scripts/nao_quality_runtime.py'), '--stop-all'],
|
||||
env=environment(value), cwd=ROOT, capture_output=True, timeout=25)
|
||||
if any(info['name'] == 'mysql' and matches(info) for info in children):
|
||||
env = environment(value)
|
||||
env['MYSQL_PWD'] = value['rootPassword']
|
||||
with contextlib.suppress(OSError, subprocess.SubprocessError):
|
||||
subprocess.run(mysql_command(value, ADMIN) + ['shutdown'], env=env,
|
||||
capture_output=True, timeout=60)
|
||||
# mysqladmin can return before InnoDB has finished closing files.
|
||||
for info in children:
|
||||
if info['name'] == 'mysql' and matches(info):
|
||||
with contextlib.suppress(psutil.NoSuchProcess, psutil.TimeoutExpired):
|
||||
psutil.Process(info['pid']).wait(60)
|
||||
survivors = [info for info in children if matches(info)]
|
||||
if survivors:
|
||||
state['children'] = survivors
|
||||
atomic(STATE, state)
|
||||
else:
|
||||
STATE.unlink(missing_ok=True)
|
||||
STOP.unlink(missing_ok=True)
|
||||
(PRIVATE / 'bootstrap.sql').unlink(missing_ok=True)
|
||||
os.close(lock)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action', choices=('start', 'stop', 'status', 'verify', 'serve'))
|
||||
parser.add_argument('--lan', action='store_true')
|
||||
parser.add_argument('--database-only', action='store_true', help=argparse.SUPPRESS)
|
||||
args = parser.parse_args()
|
||||
if args.action in ('start', 'serve', 'verify'):
|
||||
private_directory(PRIVATE)
|
||||
private_directory(DATA)
|
||||
(DATA / 'logs').mkdir(exist_ok=True)
|
||||
(DATA / 'sources').mkdir(exist_ok=True)
|
||||
(DATA / 'archive').mkdir(exist_ok=True)
|
||||
if args.action == 'serve':
|
||||
return serve(args.database_only)
|
||||
updater = ROOT / 'deploy/offline/update_manager.py'
|
||||
if args.action == 'start' and IS_WINDOWS and updater.is_file():
|
||||
recovered = subprocess.run([str(PYTHON), str(updater), 'recover'], cwd=ROOT,
|
||||
env=environment(configuration()), capture_output=True, text=True,
|
||||
encoding='utf-8')
|
||||
if recovered.returncode:
|
||||
try:
|
||||
message = json.loads(recovered.stdout).get('error')
|
||||
except (ValueError, AttributeError):
|
||||
message = None
|
||||
raise RuntimeError(message or '程序更新正在进行或尚未安全恢复,请查看更新记录。')
|
||||
state = read_state()
|
||||
running = matches(state.get('supervisor', {}))
|
||||
if args.action == 'status':
|
||||
print('已启动' if running and state.get('ready') else '正在启动或停止' if running else '已停止')
|
||||
return
|
||||
if args.action == 'stop':
|
||||
if not running:
|
||||
if any(matches(info) for info in state.get('children', [])):
|
||||
raise RuntimeError('发现上次退出后遗留的进程,请根据日志检查;未结束任何不明进程。')
|
||||
print('服务已停止。')
|
||||
return
|
||||
atomic(STOP, {'token': state['token']})
|
||||
for _ in range(240):
|
||||
if not matches(state['supervisor']):
|
||||
print('服务已停止。')
|
||||
return
|
||||
time.sleep(.5)
|
||||
raise RuntimeError('服务仍在安全关闭,请稍后检查状态。')
|
||||
if running:
|
||||
if args.lan and configuration()['webBind'] != '0.0.0.0':
|
||||
raise RuntimeError('请先停止服务,再使用 --lan 启动。')
|
||||
print('服务已启动。' if state.get('ready') else '服务正在启动,请稍候。')
|
||||
for url in addresses(configuration()):
|
||||
print(url)
|
||||
return
|
||||
if any(matches(info) for info in state.get('children', [])):
|
||||
raise RuntimeError('发现上次退出后的服务进程,原数据已保留;请检查日志。')
|
||||
value = verify()
|
||||
if args.action == 'verify':
|
||||
print('文件完整性、Java、MySQL、AI Python 与 AI 原生程序检查通过。')
|
||||
return
|
||||
if args.lan and value['webBind'] != '0.0.0.0':
|
||||
value['webBind'] = '0.0.0.0'
|
||||
atomic(CONFIG, value)
|
||||
with (DATA / 'logs/supervisor.log').open('ab') as log:
|
||||
options = {'creationflags': subprocess.DETACHED_PROCESS | subprocess.CREATE_NEW_PROCESS_GROUP} if IS_WINDOWS else {'start_new_session': True}
|
||||
process = subprocess.Popen([str(PYTHON), str(Path(__file__).resolve()), 'serve'], cwd=ROOT,
|
||||
env=environment(value), stdin=subprocess.DEVNULL, stdout=log, stderr=log, **options)
|
||||
for _ in range(2000):
|
||||
state = read_state()
|
||||
if state.get('ready') and state.get('supervisor', {}).get('pid') == process.pid and matches(state['supervisor']):
|
||||
for url in addresses(value):
|
||||
print(url)
|
||||
print('管理员初始登录信息:' + str(DATA / '登录信息.txt'))
|
||||
if IS_WINDOWS:
|
||||
os.startfile(addresses(value)[0])
|
||||
return
|
||||
if process.poll() is not None:
|
||||
raise RuntimeError('启动未完成,请查看 runtimedata/logs/supervisor.log。')
|
||||
time.sleep(.5)
|
||||
raise RuntimeError('服务仍在启动,请检查状态与日志。')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
main()
|
||||
except (RuntimeError, OSError, subprocess.SubprocessError) as error:
|
||||
print(str(error), file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,102 @@
|
||||
"""Serve built assets and stream the authenticated API on one LAN port."""
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import http.client
|
||||
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
HOP_HEADERS = {'connection', 'keep-alive', 'proxy-authenticate', 'proxy-authorization',
|
||||
'te', 'trailer', 'transfer-encoding', 'upgrade'}
|
||||
|
||||
|
||||
class Gateway(SimpleHTTPRequestHandler):
|
||||
protocol_version = 'HTTP/1.1'
|
||||
backend_port = 5281
|
||||
extensions_map = {**SimpleHTTPRequestHandler.extensions_map, '.js': 'text/javascript',
|
||||
'.mjs': 'text/javascript', '.css': 'text/css', '.svg': 'image/svg+xml',
|
||||
'.wasm': 'application/wasm'}
|
||||
|
||||
def log_message(self, *_):
|
||||
# Do not record query strings, authentication data or business values.
|
||||
pass
|
||||
|
||||
def _proxy(self):
|
||||
upstream = http.client.HTTPConnection('127.0.0.1', self.backend_port, timeout=1800)
|
||||
try:
|
||||
if self.headers.get('Transfer-Encoding'):
|
||||
self.send_error(400, 'Content-Length is required')
|
||||
return
|
||||
length = int(self.headers.get('Content-Length', '0'))
|
||||
if length < 0 or length > 273 * 1024 * 1024:
|
||||
self.send_error(413)
|
||||
return
|
||||
upstream.putrequest(self.command, self.path, skip_host=True, skip_accept_encoding=True)
|
||||
connection_headers = {s.strip().lower() for s in self.headers.get('Connection', '').split(',')}
|
||||
for name, value in self.headers.items():
|
||||
if name.lower() not in HOP_HEADERS | connection_headers:
|
||||
upstream.putheader(name, value)
|
||||
upstream.putheader('Connection', 'close')
|
||||
upstream.endheaders()
|
||||
remaining = length
|
||||
while remaining:
|
||||
chunk = self.rfile.read(min(remaining, 1024 * 1024))
|
||||
if not chunk:
|
||||
raise ConnectionError('Incomplete upload')
|
||||
upstream.send(chunk)
|
||||
remaining -= len(chunk)
|
||||
response = upstream.getresponse()
|
||||
self.send_response(response.status)
|
||||
response_connection = {s.strip().lower() for s in response.getheader('Connection', '').split(',')}
|
||||
for name, value in response.getheaders():
|
||||
if name.lower() not in HOP_HEADERS | response_connection:
|
||||
self.send_header(name, value)
|
||||
self.send_header('Connection', 'close')
|
||||
self.send_header('X-Content-Type-Options', 'nosniff')
|
||||
self.end_headers()
|
||||
self.close_connection = True
|
||||
if self.command != 'HEAD':
|
||||
while chunk := response.read1(64 * 1024):
|
||||
self.wfile.write(chunk)
|
||||
self.wfile.flush() # SSE events must arrive without buffering.
|
||||
except (BrokenPipeError, ConnectionResetError):
|
||||
self.close_connection = True
|
||||
except (OSError, http.client.HTTPException, ValueError):
|
||||
self.close_connection = True
|
||||
# Never expose upstream exception text or an internal file path.
|
||||
if not self.wfile.closed:
|
||||
try:
|
||||
self.send_error(502, 'Backend unavailable')
|
||||
except OSError:
|
||||
pass
|
||||
finally:
|
||||
upstream.close()
|
||||
|
||||
def _dispatch(self):
|
||||
path = urlsplit(self.path).path
|
||||
if path == '/api' or path.startswith('/api/') or path == '/actuator/health':
|
||||
return self._proxy()
|
||||
if self.command not in ('GET', 'HEAD'):
|
||||
self.send_error(405)
|
||||
return
|
||||
candidate = Path(self.translate_path(self.path)).resolve()
|
||||
root = Path(self.directory).resolve()
|
||||
if not candidate.is_relative_to(root):
|
||||
self.send_error(403)
|
||||
return
|
||||
if candidate.is_dir() and not (candidate / 'index.html').is_file():
|
||||
self.send_error(404)
|
||||
return
|
||||
if not candidate.exists() and not Path(path).suffix:
|
||||
self.path = '/index.html'
|
||||
return super().do_HEAD() if self.command == 'HEAD' else super().do_GET()
|
||||
|
||||
do_GET = do_HEAD = do_POST = do_PUT = do_PATCH = do_DELETE = do_OPTIONS = _dispatch
|
||||
|
||||
|
||||
def make_server(root, address, port, backend_port=5281):
|
||||
handler = type('ConfiguredGateway', (Gateway,), {'backend_port': backend_port})
|
||||
server = ThreadingHTTPServer((address, port), functools.partial(handler, directory=str(root)))
|
||||
server.daemon_threads = True
|
||||
return server
|
||||
@@ -0,0 +1,154 @@
|
||||
"""Portable migration validation; no source-machine paths or credentials are logged."""
|
||||
from __future__ import annotations
|
||||
|
||||
import gzip
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path, PurePosixPath, PureWindowsPath
|
||||
import re
|
||||
import tarfile
|
||||
import uuid
|
||||
|
||||
|
||||
def sha256(path):
|
||||
with Path(path).open('rb') as stream:
|
||||
return hashlib.file_digest(stream, 'sha256').hexdigest()
|
||||
|
||||
|
||||
def sql_text(value):
|
||||
return 'CONVERT(0x' + value.encode('utf-8').hex() + ' USING utf8mb4)'
|
||||
|
||||
|
||||
def relative_name(name):
|
||||
if not isinstance(name, str) or not name or '\\' in name or ':' in name:
|
||||
raise ValueError('迁移文件路径无效。')
|
||||
path = PurePosixPath(name)
|
||||
if not path.parts or path.is_absolute() or path.as_posix() != name or any(part in ('.', '..', '') for part in path.parts):
|
||||
raise ValueError('迁移文件路径无效。')
|
||||
for part in path.parts:
|
||||
if part.endswith((' ', '.')) or re.fullmatch(r'(?i)(CON|PRN|AUX|NUL|COM[1-9]|LPT[1-9])(?:\..*)?', part):
|
||||
raise ValueError('迁移文件名不兼容 Windows。')
|
||||
return path
|
||||
|
||||
|
||||
def path_under_source(stored, source_root):
|
||||
# A Windows restore must interpret the old Mac path as POSIX, not Path().
|
||||
path, root = PurePosixPath(stored), PurePosixPath(source_root)
|
||||
if not path.is_absolute() or '..' in path.parts or not path.is_relative_to(root):
|
||||
raise ValueError('来源文件引用不在已验证的归档目录内。')
|
||||
return relative_name(path.relative_to(root).as_posix()).as_posix()
|
||||
|
||||
|
||||
def extract_checked(archive, destination, expected):
|
||||
destination = Path(destination)
|
||||
destination.mkdir(parents=True, exist_ok=False, mode=0o700)
|
||||
observed = set()
|
||||
for name in expected:
|
||||
relative_name(name)
|
||||
folded = [name.casefold() for name in expected]
|
||||
if len(folded) != len(set(folded)):
|
||||
raise ValueError('迁移文件在 Windows 上存在重名。')
|
||||
with tarfile.open(archive, 'r:gz') as source:
|
||||
for member in source:
|
||||
rel = relative_name(member.name)
|
||||
if not member.isfile() or member.name in observed or member.name not in expected:
|
||||
raise ValueError('迁移归档包含未登记或无效的文件。')
|
||||
entry = expected[member.name]
|
||||
target = destination.joinpath(*rel.parts)
|
||||
target.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
if not target.resolve().is_relative_to(destination.resolve()):
|
||||
raise ValueError('迁移文件路径越界。')
|
||||
content = source.extractfile(member)
|
||||
if content is None:
|
||||
raise ValueError('迁移文件无法读取。')
|
||||
digest = hashlib.sha256()
|
||||
with content, target.open('xb') as output:
|
||||
while chunk := content.read(1024 * 1024):
|
||||
digest.update(chunk)
|
||||
output.write(chunk)
|
||||
target.chmod(0o600)
|
||||
if target.stat().st_size != entry['bytes'] or digest.hexdigest() != entry['sha256']:
|
||||
raise ValueError('迁移文件完整性检查失败。')
|
||||
observed.add(member.name)
|
||||
if observed != set(expected):
|
||||
raise ValueError('迁移归档缺少文件。')
|
||||
|
||||
|
||||
def table_counts(query):
|
||||
result = {}
|
||||
for name in query('SHOW TABLES;').splitlines():
|
||||
if not re.fullmatch(r'[A-Za-z0-9_]{1,64}', name):
|
||||
raise ValueError('数据库表名无效。')
|
||||
result[name] = int(query('SELECT COUNT(*) FROM `' + name + '`;').strip())
|
||||
return result
|
||||
|
||||
|
||||
def query_evidence(query):
|
||||
current = query("SELECT release_id FROM current_release WHERE dataset_id='quality';").strip()
|
||||
if not current:
|
||||
return {'releaseId': None, 'months': []}
|
||||
if not re.fullmatch(r'[A-Za-z0-9-]{1,80}', current):
|
||||
raise ValueError('正式数据版本无效。')
|
||||
result = {'releaseId': current, 'months': []}
|
||||
components = ['black_responsibility_scrap', 'black_material_scrap', 'black_external_scrap',
|
||||
'bright_headquarters_scrap', 'bright_plating_scrap', 'bright_material_scrap',
|
||||
'bright_processing_scrap', 'bright_diffusion_scrap']
|
||||
for stage in ('black', 'bright'):
|
||||
pairs = "'stage','" + stage + "','month',f.month_key,'rows',COUNT(*),'qualified',SUM(f.qualified_weight),'scrap',SUM(f.scrap_weight),'total',SUM(f.total_weight)"
|
||||
for key in components:
|
||||
pairs += ", '" + key + "',SUM(CAST(NULLIF(JSON_UNQUOTE(JSON_EXTRACT(f.components,'$." + key + "')),'null') AS DECIMAL(28,10)))"
|
||||
result['months'].extend(json.loads(line) for line in query("SELECT JSON_OBJECT(" + pairs + ") FROM fact_" + stage + " f WHERE EXISTS(SELECT 1 FROM release_member rm WHERE rm.release_id='" + current + "' AND rm.transform_build_id=f.build_id) GROUP BY f.month_key ORDER BY f.month_key;").splitlines())
|
||||
result['members'] = [json.loads(line) for line in query("SELECT JSON_OBJECT('sourceDocumentId',m.source_document_id,'sourceRevisionId',m.source_revision_id,'buildId',m.transform_build_id,'sourceSha256',r.sha256) FROM release_member m JOIN source_revision r ON r.id=m.source_revision_id WHERE m.release_id='" + current + "' ORDER BY m.source_document_id;").splitlines()]
|
||||
return result
|
||||
|
||||
|
||||
_VALUES_TOKENS = re.compile(rb"'(?:[^'\\]+|\\.)*'|(\),\()")
|
||||
|
||||
|
||||
def inserted_rows(line):
|
||||
if not line.startswith(b'INSERT INTO `') or b' VALUES (' not in line:
|
||||
raise ValueError('不支持的数据库导出行。')
|
||||
values = line.split(b' VALUES ', 1)[1]
|
||||
if not values.rstrip().endswith(b';'):
|
||||
raise ValueError('数据库导出行不完整。')
|
||||
return 1 + sum(match[1] is not None for match in _VALUES_TOKENS.finditer(values))
|
||||
|
||||
|
||||
def dump_counts(path):
|
||||
counts, uncompressed = {}, 0
|
||||
with gzip.open(path, 'rb') as source:
|
||||
for line in source:
|
||||
uncompressed += len(line)
|
||||
create = re.match(rb'^CREATE TABLE `([A-Za-z0-9_]+)`', line)
|
||||
if create:
|
||||
counts[create[1].decode()] = 0
|
||||
insert = re.match(rb'^INSERT INTO `([A-Za-z0-9_]+)`', line)
|
||||
if insert:
|
||||
name = insert[1].decode()
|
||||
if name not in counts:
|
||||
raise ValueError('导出数据缺少表结构。')
|
||||
counts[name] += inserted_rows(line)
|
||||
return counts, uncompressed
|
||||
|
||||
|
||||
def filter_models(value):
|
||||
profiles = value.get('profiles')
|
||||
if not isinstance(profiles, list):
|
||||
profiles = [{**value, 'id': 'migrated-profile', 'name': '迁移模型'}]
|
||||
def excluded(profile):
|
||||
return any('mimo' in str(profile.get(key, '')).lower() for key in ('model', 'name', 'baseUrl'))
|
||||
kept = [profile.copy() for profile in profiles if not excluded(profile)]
|
||||
removed = [profile for profile in profiles if excluded(profile)]
|
||||
if not kept:
|
||||
return None, removed
|
||||
selected = next((profile for profile in kept if profile.get('id') == value.get('activeProfileId')), kept[0])
|
||||
required = ('baseUrl', 'apiKey', 'model')
|
||||
if any(not isinstance(selected.get(key), str) or not selected[key] for key in required):
|
||||
raise ValueError('需要迁移的模型配置不完整。')
|
||||
filtered = {key: value[key] for key in ('schemaVersion',) if key in value}
|
||||
filtered.update(selected)
|
||||
filtered.pop('id', None)
|
||||
filtered.pop('name', None)
|
||||
filtered.update({'schemaVersion': 3, 'profiles': kept, 'activeProfileId': selected['id']})
|
||||
filtered['revision'] = value.get('revision') or str(uuid.uuid4())
|
||||
return filtered, removed
|
||||
@@ -0,0 +1,200 @@
|
||||
"""Capture verified source files and private model settings for a Windows migration."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import sqlite3
|
||||
import sys
|
||||
import tarfile
|
||||
import uuid
|
||||
|
||||
from migration_common import filter_models, relative_name, sha256
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / 'scripts'))
|
||||
import backup
|
||||
|
||||
|
||||
def private_json(path, data):
|
||||
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, 'w', encoding='utf-8') as output:
|
||||
json.dump(data, output, ensure_ascii=False, indent=2)
|
||||
output.write('\n')
|
||||
|
||||
|
||||
def native_copy(source, target):
|
||||
source_db = sqlite3.connect(source.resolve().as_uri() + '?mode=ro', uri=True)
|
||||
destination = sqlite3.connect(target)
|
||||
try:
|
||||
source_db.backup(destination)
|
||||
destination.execute('PRAGMA foreign_keys=OFF')
|
||||
destination.execute('PRAGMA secure_delete=ON')
|
||||
tables = {row[0] for row in destination.execute("SELECT name FROM sqlite_master WHERE type='table'")}
|
||||
# Access sessions, external integrations and cached provider keys are
|
||||
# recreated on Windows. Historical chats and their ownership stay intact.
|
||||
for name in ('session', 'verification', 'api_key', 'jwks', 'oauth_access_token',
|
||||
'oauth_refresh_token', 'oauth_consent', 'oauth_client', 'mcp_user_token',
|
||||
'mcp_oauth_client', 'sandbox_secret', 'log', 'mcp_call_log'):
|
||||
if name in tables:
|
||||
destination.execute('DELETE FROM "' + name + '"')
|
||||
if 'project_llm_config' in tables:
|
||||
destination.execute('DELETE FROM project_llm_config')
|
||||
for name in ('user', 'account', 'organization'):
|
||||
if name not in tables:
|
||||
continue
|
||||
columns = list(destination.execute('PRAGMA table_info("' + name + '")'))
|
||||
for column in columns:
|
||||
key = column[1]
|
||||
if key in ('github_access_token', 'gitlab_access_token', 'access_token', 'refresh_token',
|
||||
'id_token', 'google_client_secret'):
|
||||
replacement = "''" if column[3] else 'NULL'
|
||||
destination.execute('UPDATE "' + name + '" SET "' + key + '"=' + replacement)
|
||||
destination.commit()
|
||||
destination.execute('VACUUM')
|
||||
assert destination.execute('PRAGMA integrity_check').fetchone()[0] == 'ok'
|
||||
finally:
|
||||
destination.close()
|
||||
source_db.close()
|
||||
target.chmod(0o600)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--work', type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
work = args.work.resolve()
|
||||
payload = work / 'migration'
|
||||
manifest = json.loads((payload / 'source-snapshot.json').read_text())
|
||||
database = manifest['sourceDatabase']
|
||||
query = lambda sql: backup.client(sql, database)
|
||||
archive_root = ROOT / 'runtimedata/archive'
|
||||
files, sources, digest_index = {}, {}, {}
|
||||
for path in sorted(archive_root.rglob('*')):
|
||||
if path.is_symlink():
|
||||
raise RuntimeError('来源归档包含符号链接。')
|
||||
if not path.is_file():
|
||||
continue
|
||||
rel = relative_name(path.relative_to(archive_root).as_posix()).as_posix()
|
||||
digest = sha256(path)
|
||||
files[rel] = {'sha256': digest, 'bytes': path.stat().st_size}
|
||||
sources[rel] = path
|
||||
digest_index.setdefault(digest, rel)
|
||||
bindings = []
|
||||
missing_optional = 0
|
||||
for table, column, state in (
|
||||
('source_revision', 'archive_path', "'REQUIRED'"),
|
||||
('source_candidate', 'source_path', 'status'),
|
||||
('source_candidate', 'archive_path', 'status'),
|
||||
('source_upload_draft', 'source_path', "IF(expires_at<CURRENT_TIMESTAMP(6),'EXPIRED','ACTIVE')"),
|
||||
):
|
||||
rows = query("SELECT JSON_OBJECT('id',id,'path'," + column + ",'sha256',sha256,'size',file_size,'state'," + state + ') FROM ' + table + ' WHERE ' + column + ' IS NOT NULL;')
|
||||
for line in rows.splitlines():
|
||||
row = json.loads(line)
|
||||
old = Path(row['path'])
|
||||
digest = row['sha256']
|
||||
relative = None
|
||||
if old.is_relative_to(archive_root) and old.is_file():
|
||||
relative = old.relative_to(archive_root).as_posix()
|
||||
elif digest in digest_index:
|
||||
relative = digest_index[digest]
|
||||
elif old.is_file() and not old.is_symlink():
|
||||
if sha256(old) != digest or old.stat().st_size != int(row['size']):
|
||||
raise RuntimeError('需要迁移的来源文件与登记内容不一致。')
|
||||
relative = 'migration-inputs/' + digest + '.xlsx'
|
||||
files[relative] = {'sha256': digest, 'bytes': old.stat().st_size}
|
||||
sources[relative] = old
|
||||
digest_index[digest] = relative
|
||||
optional = False
|
||||
if relative is None:
|
||||
if table != 'source_upload_draft' or row['state'] != 'EXPIRED':
|
||||
raise RuntimeError('需要迁移的来源文件缺失。')
|
||||
relative = 'expired-upload-drafts/' + row['id'] + '.xlsx'
|
||||
optional = True
|
||||
missing_optional += 1
|
||||
else:
|
||||
info = files[relative]
|
||||
if info['sha256'] != digest or info['bytes'] != int(row['size']):
|
||||
raise RuntimeError('来源引用与归档校验不一致。')
|
||||
bindings.append({'table': table, 'column': column, 'id': row['id'], 'oldPath': row['path'],
|
||||
'relative': relative, 'sha256': digest, 'bytes': int(row['size']), 'expiredMissing': optional})
|
||||
archive = payload / 'archive.tar.gz'
|
||||
with tarfile.open(archive, 'w:gz', compresslevel=1) as output:
|
||||
for rel, path in sorted(sources.items()):
|
||||
output.add(path, arcname=rel, recursive=False)
|
||||
if sha256(path) != files[rel]['sha256']:
|
||||
raise RuntimeError('打包期间来源文件发生变化。')
|
||||
archive.chmod(0o600)
|
||||
print('Source files captured:', len(files), 'references:', len(bindings), flush=True)
|
||||
|
||||
config = json.loads((ROOT / '.runtime/nao-model-config.json').read_text())
|
||||
filtered, excluded = filter_models(config)
|
||||
if filtered:
|
||||
private_json(payload / 'model-config.json', filtered)
|
||||
kept_keys = {p.get('apiKey') for p in (filtered or {}).get('profiles', [])}
|
||||
forbidden_keys = {p.get('apiKey') for p in excluded if p.get('apiKey') and p.get('apiKey') not in kept_keys}
|
||||
# Used only by the local privacy verifier; never shipped or printed.
|
||||
private_json(work / 'excluded-model-audit-private.json', {'keys': sorted(forbidden_keys)})
|
||||
|
||||
session_rows = [json.loads(line) for line in query("SELECT JSON_OBJECT('id',id,'userId',user_id) FROM analysis_session;").splitlines()]
|
||||
users = {row['id'] for row in manifest['sourceUsers']}
|
||||
native_source = ROOT / '.runtime/nao-users'
|
||||
native_stage = work / 'native-snapshot'
|
||||
native_stage.mkdir(mode=0o700, exist_ok=False)
|
||||
runtime_roots = []
|
||||
for actor in users:
|
||||
if not re.fullmatch(r'[a-fA-F0-9-]{36}', actor):
|
||||
raise RuntimeError('用户标识无效。')
|
||||
if (native_source / actor / 'private/nao.sqlite').is_file():
|
||||
runtime_roots.append(native_source / actor)
|
||||
for row in session_rows:
|
||||
source = native_source / row['userId'] / 'sessions' / row['id']
|
||||
if (source / 'private/nao.sqlite').is_file():
|
||||
runtime_roots.append(source)
|
||||
selected = set()
|
||||
for runtime in runtime_roots:
|
||||
for name in ('credentials.json', 'nao.sqlite', 'native-chat.json', 'current-session.json', 'sessions.json'):
|
||||
source = runtime / 'private' / name
|
||||
if source.is_file() and not source.is_symlink():
|
||||
selected.add(source)
|
||||
evidence = runtime / 'evidence'
|
||||
if evidence.is_dir():
|
||||
selected.update(p for p in evidence.rglob('*') if p.is_file() and not p.is_symlink())
|
||||
native_files = {}
|
||||
for source in sorted(selected):
|
||||
rel = source.relative_to(native_source).as_posix()
|
||||
relative_name(rel)
|
||||
target = native_stage / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
if source.name == 'nao.sqlite':
|
||||
native_copy(source, target)
|
||||
else:
|
||||
shutil.copy2(source, target)
|
||||
target.chmod(0o600)
|
||||
content = target.read_bytes()
|
||||
if any(key.encode() in content for key in forbidden_keys):
|
||||
raise RuntimeError('原生会话中检测到被排除模型的密钥,已停止打包。')
|
||||
native_files[rel] = {'sha256': sha256(target), 'bytes': target.stat().st_size}
|
||||
with tarfile.open(payload / 'native-history.tar.gz', 'w:gz', compresslevel=1) as output:
|
||||
for rel in sorted(native_files):
|
||||
output.add(native_stage / rel, arcname=rel, recursive=False)
|
||||
(payload / 'native-history.tar.gz').chmod(0o600)
|
||||
manifest.update({'migrationId': uuid.uuid4().hex, 'archiveFiles': files, 'pathBindings': bindings,
|
||||
'archiveSha256': sha256(archive), 'nativeHistoryFiles': native_files,
|
||||
'nativeHistorySha256': sha256(payload / 'native-history.tar.gz'),
|
||||
'sourceProjectRoot': str(ROOT), 'modelProfilesMigrated': len((filtered or {}).get('profiles', [])),
|
||||
'modelProfilesExcluded': len(excluded), 'modelConfigSha256': sha256(payload / 'model-config.json') if filtered else None,
|
||||
'expiredMissingDrafts': missing_optional,
|
||||
'sourceDatabaseBytes': int(backup.client("SELECT SUM(data_length+index_length) FROM information_schema.tables WHERE table_schema='" + database + "';").strip()),
|
||||
'nativeHistoryDatabases': len(runtime_roots)})
|
||||
private_json(work / 'migration-file-snapshot.json', manifest)
|
||||
print('Model profiles migrated:', manifest['modelProfilesMigrated'], '; MiMo excluded:', len(excluded), flush=True)
|
||||
print('Native business-history databases:', len(runtime_roots), '; files:', len(native_files), flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,288 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build a signed, program-only Windows update bundle without network access."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
from dataclasses import dataclass
|
||||
import hashlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import stat
|
||||
import tempfile
|
||||
import zipfile
|
||||
|
||||
from cryptography.hazmat.primitives import serialization
|
||||
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey, Ed25519PublicKey
|
||||
|
||||
import update_manager
|
||||
|
||||
|
||||
VERSION = re.compile(r'[A-Za-z0-9][A-Za-z0-9._-]{0,79}')
|
||||
ARCHIVE_VERSION = re.compile(r'[A-Za-z0-9][A-Za-z0-9._-]{0,100}')
|
||||
REQUIRED_PROGRAM_FILES = {
|
||||
'app/backend.jar',
|
||||
'deploy/offline/control.py',
|
||||
'deploy/offline/update_manager.py',
|
||||
'backend/src/main/resources/analysis-runtime-policy.json',
|
||||
}
|
||||
SENSITIVE_PARTS = {'.runtime', 'runtimedata', 'migration', 'private', 'secrets', 'credentials'}
|
||||
SENSITIVE_SUFFIXES = {'.pem', '.key', '.p12', '.pfx', '.jks'}
|
||||
|
||||
|
||||
class PublishError(RuntimeError):
|
||||
"""Release inputs cannot produce a safe update package."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PublishedArtifacts:
|
||||
archive: Path
|
||||
latest: Path
|
||||
file_count: int
|
||||
archive_sha256: str
|
||||
|
||||
|
||||
def _manifest(root: Path) -> dict:
|
||||
try:
|
||||
value = update_manager.read(root / 'manifest.json')
|
||||
except Exception as error:
|
||||
raise PublishError('无法读取部署包 manifest.json。') from error
|
||||
if not isinstance(value, dict) or value.get('platform') != 'windows' or value.get('architecture') != 'x86_64':
|
||||
raise PublishError('输入目录必须是 Windows x64 离线部署包。')
|
||||
if not isinstance(value.get('files'), list) or len(value['files']) > 50000:
|
||||
raise PublishError('部署包文件清单无效。')
|
||||
for key in ('runtimes', 'pythonWheels'):
|
||||
if not isinstance(value.get(key), list):
|
||||
raise PublishError('部署包缺少运行环境指纹信息。')
|
||||
current_sequence = value.get('updateSequence', 0)
|
||||
if type(current_sequence) is not int or current_sequence < 0:
|
||||
raise PublishError('部署包更新序号无效。')
|
||||
return value
|
||||
|
||||
|
||||
def _sensitive_path(name: str) -> bool:
|
||||
path = update_manager.safe_path(name)
|
||||
parts = [part.casefold() for part in path.parts]
|
||||
return (name == update_manager.KEY
|
||||
or name.startswith('deploy/bootstrap/')
|
||||
or any(part in SENSITIVE_PARTS for part in parts)
|
||||
or path.suffix.casefold() in SENSITIVE_SUFFIXES)
|
||||
|
||||
|
||||
def _program_inventory(root: Path, manifest: dict) -> list[dict]:
|
||||
entries: dict[str, dict] = {}
|
||||
folded: set[str] = set()
|
||||
for entry in manifest['files']:
|
||||
if not isinstance(entry, dict):
|
||||
raise PublishError('部署包文件清单无效。')
|
||||
name = entry.get('path')
|
||||
try:
|
||||
update_manager.safe_path(name)
|
||||
except update_manager.UpdateError as error:
|
||||
raise PublishError('部署包文件清单含有无效路径。') from error
|
||||
if name.casefold() in folded:
|
||||
raise PublishError('部署包文件清单含有重复或大小写冲突路径。')
|
||||
folded.add(name.casefold())
|
||||
if not update_manager.program_path(name):
|
||||
continue
|
||||
if _sensitive_path(name):
|
||||
raise PublishError('程序清单中包含受保护的私有、迁移、引导或密钥文件路径。')
|
||||
if entry.get('symlink') is not None or not update_manager.SHA.fullmatch(str(entry.get('sha256', ''))):
|
||||
raise PublishError('程序文件清单缺少有效摘要或包含符号链接。')
|
||||
path = update_manager.destination(root, name)
|
||||
if not path.is_file() or path.is_symlink():
|
||||
raise PublishError('程序清单文件缺失或不是普通文件。')
|
||||
try:
|
||||
digest = update_manager.digest(path)
|
||||
except OSError as error:
|
||||
raise PublishError('读取程序清单文件失败。') from error
|
||||
if digest != entry['sha256']:
|
||||
raise PublishError('部署包程序文件与 manifest.json 摘要不一致。')
|
||||
entries[name] = {'path': name, 'sha256': digest, 'size': path.stat().st_size}
|
||||
|
||||
if not REQUIRED_PROGRAM_FILES.issubset(entries) or not any(name.startswith('app/web/') for name in entries):
|
||||
raise PublishError('程序清单缺少更新所需的后端、网页、策略或更新管理程序。')
|
||||
|
||||
# The updater replaces program files as a complete snapshot. Refuse to
|
||||
# sign an incomplete manifest if a program file exists outside its list.
|
||||
present: set[str] = set()
|
||||
for path in root.rglob('*'):
|
||||
name = path.relative_to(root).as_posix()
|
||||
if not update_manager.program_path(name):
|
||||
continue
|
||||
if path.is_symlink():
|
||||
raise PublishError('程序目录含有符号链接或特殊文件。')
|
||||
if path.is_dir():
|
||||
continue
|
||||
if not path.is_file():
|
||||
raise PublishError('程序目录含有符号链接或特殊文件。')
|
||||
present.add(name)
|
||||
if present != set(entries):
|
||||
raise PublishError('manifest.json 与磁盘上的完整程序文件清单不一致。')
|
||||
return [entries[name] for name in sorted(entries)]
|
||||
|
||||
|
||||
def _load_private_key(path: Path, root: Path, output_dir: Path) -> Ed25519PrivateKey:
|
||||
try:
|
||||
resolved = path.expanduser().resolve(strict=True)
|
||||
if resolved.is_relative_to(root) or resolved.is_relative_to(output_dir):
|
||||
raise PublishError('签名私钥必须保存在部署包和更新产物目录之外。')
|
||||
key = serialization.load_pem_private_key(resolved.read_bytes(), password=None)
|
||||
except PublishError:
|
||||
raise
|
||||
except Exception as error:
|
||||
raise PublishError('无法读取未加密 PEM 格式的 Ed25519 签名私钥。') from error
|
||||
if not isinstance(key, Ed25519PrivateKey):
|
||||
raise PublishError('签名私钥必须是 Ed25519。')
|
||||
return key
|
||||
|
||||
|
||||
def _write_archive(root: Path, archive_temp: Path, files: list[dict]) -> None:
|
||||
with zipfile.ZipFile(archive_temp, 'w', compression=zipfile.ZIP_DEFLATED, compresslevel=9, allowZip64=True) as output:
|
||||
for entry in files:
|
||||
name = entry['path']
|
||||
source = update_manager.destination(root, name)
|
||||
info = zipfile.ZipInfo(name, date_time=(1980, 1, 1, 0, 0, 0))
|
||||
info.compress_type = zipfile.ZIP_DEFLATED
|
||||
info.create_system = 3
|
||||
info.external_attr = (stat.S_IFREG | 0o644) << 16
|
||||
digest = hashlib.sha256()
|
||||
size = 0
|
||||
with source.open('rb') as reader, output.open(info, 'w') as writer:
|
||||
while block := reader.read(1024 * 1024):
|
||||
size += len(block)
|
||||
if size > entry['size'] or size > update_manager.MAX_EXPANDED:
|
||||
raise PublishError('程序文件在打包期间发生变化。')
|
||||
digest.update(block)
|
||||
writer.write(block)
|
||||
if size != entry['size'] or digest.hexdigest() != entry['sha256']:
|
||||
raise PublishError('程序文件在打包期间发生变化,已停止发布。')
|
||||
|
||||
|
||||
def publish(root: Path, version: str, sequence: int, private_key_path: Path, output_dir: Path,
|
||||
allow_baseline: bool = False) -> PublishedArtifacts:
|
||||
"""Create quality-update-<version>.zip and signed latest.json locally."""
|
||||
root = root.expanduser().resolve(strict=True)
|
||||
if not root.is_dir():
|
||||
raise PublishError('部署包根路径必须是目录。')
|
||||
if not isinstance(version, str) or not VERSION.fullmatch(version):
|
||||
raise PublishError('版本号只能包含字母、数字、点、下划线和连字符。')
|
||||
if not ARCHIVE_VERSION.fullmatch(version):
|
||||
raise PublishError('版本号不能用于更新文件名。')
|
||||
if type(sequence) is not int or sequence <= 0:
|
||||
raise PublishError('更新序号必须是正整数。')
|
||||
|
||||
manifest = _manifest(root)
|
||||
current_sequence = manifest['updateSequence']
|
||||
if sequence < current_sequence:
|
||||
raise PublishError('更新序号不能低于部署包当前序号。')
|
||||
if sequence == current_sequence and not allow_baseline:
|
||||
raise PublishError('更新序号必须大于部署包当前序号。')
|
||||
if sequence == current_sequence and version != manifest.get('updateVersion'):
|
||||
raise PublishError('基线版本必须与部署包当前 updateVersion 完全一致。')
|
||||
files = _program_inventory(root, manifest)
|
||||
if sum(entry['size'] for entry in files) > update_manager.MAX_EXPANDED:
|
||||
raise PublishError('更新程序文件总大小超过限制。')
|
||||
|
||||
output_dir = output_dir.expanduser().resolve()
|
||||
if output_dir.is_relative_to(root):
|
||||
raise PublishError('更新产物目录必须位于部署包目录之外。')
|
||||
private_key = _load_private_key(Path(private_key_path), root, output_dir)
|
||||
public_key_path = update_manager.destination(root, update_manager.KEY)
|
||||
try:
|
||||
public_key = serialization.load_pem_public_key(public_key_path.read_bytes())
|
||||
except Exception as error:
|
||||
raise PublishError('无法读取部署包中的更新公钥。') from error
|
||||
if not isinstance(public_key, Ed25519PublicKey) or public_key.public_bytes(
|
||||
serialization.Encoding.Raw, serialization.PublicFormat.Raw) != private_key.public_key().public_bytes(
|
||||
serialization.Encoding.Raw, serialization.PublicFormat.Raw):
|
||||
raise PublishError('签名私钥与部署包中的更新公钥不匹配。')
|
||||
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
archive_name = f'quality-update-{version}.zip'
|
||||
archive_path = output_dir / archive_name
|
||||
latest_path = output_dir / 'latest.json'
|
||||
if archive_path.exists() or latest_path.exists():
|
||||
raise PublishError('输出目录已包含同名发布文件;请使用新的空输出目录。')
|
||||
|
||||
archive_temp: Path | None = None
|
||||
latest_temp: Path | None = None
|
||||
published_archive = False
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(prefix='.quality-update-', suffix='.tmp', dir=output_dir, delete=False) as stream:
|
||||
archive_temp = Path(stream.name)
|
||||
_write_archive(root, archive_temp, files)
|
||||
archive_size = archive_temp.stat().st_size
|
||||
if not 0 < archive_size <= update_manager.MAX_ARCHIVE:
|
||||
raise PublishError('更新压缩包大小超过限制。')
|
||||
archive_hash = update_manager.digest(archive_temp)
|
||||
release = {
|
||||
'schema': 1,
|
||||
'version': version,
|
||||
'sequence': sequence,
|
||||
'platform': 'windows-x64',
|
||||
'runtimeFingerprint': update_manager.runtime_fingerprint(manifest),
|
||||
'archive': {'file': archive_name, 'size': archive_size, 'sha256': archive_hash},
|
||||
'files': files,
|
||||
}
|
||||
raw = update_manager.canonical(release)
|
||||
if len(raw) > update_manager.MAX_MANIFEST:
|
||||
raise PublishError('更新文件清单超过大小限制。')
|
||||
envelope = {
|
||||
'payload': base64.b64encode(raw).decode('ascii'),
|
||||
'signature': base64.b64encode(private_key.sign(raw)).decode('ascii'),
|
||||
}
|
||||
try:
|
||||
update_manager.validate_release(envelope, root)
|
||||
except update_manager.UpdateError as error:
|
||||
raise PublishError('生成的更新包未通过客户端契约校验:' + str(error)) from error
|
||||
latest_bytes = update_manager.canonical(envelope)
|
||||
with tempfile.NamedTemporaryFile(prefix='.latest-', suffix='.tmp', dir=output_dir, delete=False) as stream:
|
||||
latest_temp = Path(stream.name)
|
||||
stream.write(latest_bytes)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.link(archive_temp, archive_path)
|
||||
archive_temp.unlink()
|
||||
archive_temp = None
|
||||
published_archive = True
|
||||
os.link(latest_temp, latest_path)
|
||||
latest_temp.unlink()
|
||||
latest_temp = None
|
||||
return PublishedArtifacts(archive_path, latest_path, len(files), archive_hash)
|
||||
except FileExistsError as error:
|
||||
raise PublishError('输出目录出现同名发布文件;请使用新的空输出目录。') from error
|
||||
except OSError as error:
|
||||
raise PublishError('写入更新产物失败。') from error
|
||||
finally:
|
||||
if archive_temp is not None:
|
||||
archive_temp.unlink(missing_ok=True)
|
||||
if latest_temp is not None:
|
||||
latest_temp.unlink(missing_ok=True)
|
||||
if published_archive and not latest_path.exists():
|
||||
archive_path.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--root', type=Path, required=True, help='Windows x64 offline bundle directory')
|
||||
parser.add_argument('--version', required=True, help='Release version, for example 2026.11.04.2')
|
||||
parser.add_argument('--sequence', type=int, required=True, help='Monotonically increasing update sequence')
|
||||
parser.add_argument('--private-key', type=Path, required=True, help='Ed25519 PEM private key kept outside the bundle')
|
||||
parser.add_argument('--output-dir', type=Path, required=True, help='New output directory for the ZIP and latest.json')
|
||||
parser.add_argument('--allow-baseline', action='store_true',
|
||||
help='Allow a seed release only when version and sequence exactly match the bundle manifest')
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
result = publish(args.root, args.version, args.sequence, args.private_key, args.output_dir, args.allow_baseline)
|
||||
except (PublishError, update_manager.UpdateError) as error:
|
||||
parser.error(str(error))
|
||||
print(f'已生成更新包:{result.archive}')
|
||||
print(f'已生成签名清单:{result.latest}')
|
||||
print(f'版本 {args.version},序号 {args.sequence},程序文件 {result.file_count} 个,SHA-256 {result.archive_sha256}')
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,327 @@
|
||||
"""Restore into a new database, validate, then switch the Windows deployment."""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
import gzip
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path, PurePosixPath
|
||||
import re
|
||||
import shutil
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
|
||||
import control
|
||||
from migration_common import extract_checked, query_evidence, relative_name, sha256, sql_text, table_counts
|
||||
|
||||
ROOT = control.ROOT
|
||||
PAYLOAD = ROOT / 'migration'
|
||||
JOURNAL = control.PRIVATE / 'migration-progress.json'
|
||||
RECEIPT = control.PRIVATE / 'migration-receipt.json'
|
||||
|
||||
|
||||
def checked_local(relative):
|
||||
rel = relative_name(relative)
|
||||
path = ROOT.joinpath(*rel.parts)
|
||||
if not path.resolve().is_relative_to(ROOT.resolve()):
|
||||
raise RuntimeError('恢复暂存路径越界。')
|
||||
return path
|
||||
|
||||
|
||||
def run_query(value, statement, database=None):
|
||||
command = control.mysql_command(value)
|
||||
command += ['--batch', '--skip-column-names', '--default-character-set=utf8mb4', '--max-allowed-packet=1G']
|
||||
if database:
|
||||
if not re.fullmatch(r'quality_(?:platform|restore_[a-f0-9]{12,32})', database):
|
||||
raise RuntimeError('恢复数据库名称无效。')
|
||||
command += ['--database=' + database]
|
||||
env = control.environment(value)
|
||||
env['MYSQL_PWD'] = value['rootPassword']
|
||||
result = subprocess.run(command, input=statement, text=True, encoding='utf-8',
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env, timeout=1800)
|
||||
if result.returncode:
|
||||
raise RuntimeError('恢复数据库校验未完成;已有数据库保留。')
|
||||
return result.stdout
|
||||
|
||||
|
||||
def stop_owned_service():
|
||||
state = control.read_state()
|
||||
if control.matches(state.get('supervisor', {})):
|
||||
control.atomic(control.STOP, {'token': state['token']})
|
||||
for _ in range(300):
|
||||
if not control.matches(state['supervisor']):
|
||||
break
|
||||
time.sleep(.5)
|
||||
else:
|
||||
raise RuntimeError('原服务仍在关闭,请稍后重新运行。')
|
||||
if any(control.matches(info) for info in state.get('children', [])):
|
||||
raise RuntimeError('发现本目录尚未关闭的后台进程,请检查服务日志。')
|
||||
|
||||
|
||||
def start_database(value):
|
||||
with (control.DATA / 'logs/migration-database-supervisor.log').open('ab') as log:
|
||||
options = ({'creationflags': subprocess.DETACHED_PROCESS | subprocess.CREATE_NEW_PROCESS_GROUP}
|
||||
if control.IS_WINDOWS else {'start_new_session': True})
|
||||
process = subprocess.Popen([str(control.PYTHON), str(Path(control.__file__).resolve()),
|
||||
'serve', '--database-only'], cwd=ROOT, env=control.environment(value),
|
||||
stdin=subprocess.DEVNULL, stdout=log, stderr=log, **options)
|
||||
for _ in range(360):
|
||||
state = control.read_state()
|
||||
if (state.get('ready') and state.get('databaseOnly') and
|
||||
state.get('supervisor', {}).get('pid') == process.pid and control.matches(state['supervisor'])):
|
||||
return
|
||||
if process.poll() is not None:
|
||||
raise RuntimeError('恢复数据库无法启动,请查看 runtimedata/logs。')
|
||||
time.sleep(.5)
|
||||
raise RuntimeError('恢复数据库仍未就绪,请查看日志后重试。')
|
||||
|
||||
|
||||
def restore_dump(value, database, dump):
|
||||
env = control.environment(value)
|
||||
env['MYSQL_PWD'] = value['rootPassword']
|
||||
command = control.mysql_command(value) + ['--database=' + database,
|
||||
'--default-character-set=utf8mb4', '--max-allowed-packet=1G',
|
||||
'--init-command=SET SESSION autocommit=0']
|
||||
with (control.DATA / 'logs/migration-import.log').open('ab') as errors:
|
||||
process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=errors, env=env)
|
||||
try:
|
||||
total, next_progress = dump.stat().st_size, 0
|
||||
with dump.open('rb') as raw, gzip.GzipFile(fileobj=raw) as source:
|
||||
while chunk := source.read(1024 * 1024):
|
||||
process.stdin.write(chunk)
|
||||
percent = min(99, raw.tell() * 100 // total)
|
||||
if percent >= next_progress:
|
||||
print(f'数据库导入进度约 {percent}%……', flush=True)
|
||||
next_progress = percent + 10
|
||||
process.stdin.write(b'\nCOMMIT;\n')
|
||||
process.stdin.close()
|
||||
if process.wait(timeout=1800):
|
||||
raise RuntimeError('业务数据导入失败,请查看私有恢复日志;原数据库未覆盖。')
|
||||
except BaseException:
|
||||
if process.poll() is None:
|
||||
process.kill()
|
||||
process.wait()
|
||||
raise
|
||||
|
||||
|
||||
def rebind_sources(query, manifest, archive):
|
||||
counts = {'source_revision': 0, 'source_candidate': 0, 'source_upload_draft': 0}
|
||||
for entry in manifest['pathBindings']:
|
||||
pair = (entry['table'], entry['column'])
|
||||
if pair not in {('source_revision', 'archive_path'), ('source_candidate', 'source_path'),
|
||||
('source_candidate', 'archive_path'), ('source_upload_draft', 'source_path')}:
|
||||
raise RuntimeError('来源路径绑定类型无效。')
|
||||
rel = relative_name(entry['relative'])
|
||||
target = archive.joinpath(*rel.parts)
|
||||
if not entry['expiredMissing']:
|
||||
if not target.is_file() or target.stat().st_size != entry['bytes'] or sha256(target) != entry['sha256']:
|
||||
raise RuntimeError('来源文件校验失败,恢复未切换。')
|
||||
query('UPDATE `' + entry['table'] + '` SET `' + entry['column'] + '`=' + sql_text(str(target))
|
||||
+ ' WHERE id=' + sql_text(entry['id']) + ' AND `' + entry['column'] + '`=' + sql_text(entry['oldPath']) + ';')
|
||||
actual = query('SELECT `' + entry['column'] + '` FROM `' + entry['table'] + '` WHERE id=' + sql_text(entry['id']) + ';').strip()
|
||||
if actual != str(target):
|
||||
raise RuntimeError('来源文件路径重绑定失败。')
|
||||
counts[entry['table']] += 1
|
||||
return counts
|
||||
|
||||
|
||||
def relocate_native(stage, source_root):
|
||||
source_root = PurePosixPath(source_root)
|
||||
def relocated(value):
|
||||
if isinstance(value, str) and value.startswith(str(source_root) + '/'):
|
||||
relative = PurePosixPath(value).relative_to(source_root)
|
||||
if '..' in relative.parts:
|
||||
raise RuntimeError('原生历史路径无效。')
|
||||
return str(ROOT.joinpath(*relative.parts))
|
||||
if isinstance(value, dict):
|
||||
return {key: relocated(child) for key, child in value.items()}
|
||||
if isinstance(value, list):
|
||||
return [relocated(child) for child in value]
|
||||
return value
|
||||
for path in stage.rglob('*.json'):
|
||||
control.atomic(path, relocated(json.loads(path.read_text(encoding='utf-8'))))
|
||||
databases = 0
|
||||
for path in stage.rglob('nao.sqlite'):
|
||||
connection = sqlite3.connect(path)
|
||||
try:
|
||||
connection.execute('PRAGMA secure_delete=ON')
|
||||
for table, column in (('project', 'path'), ('message_part', 'storage_path')):
|
||||
exists = connection.execute("SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", (table,)).fetchone()
|
||||
if not exists:
|
||||
continue
|
||||
for rowid, original in connection.execute('SELECT rowid,"' + column + '" FROM "' + table + '" WHERE "' + column + '" IS NOT NULL').fetchall():
|
||||
replacement = relocated(original)
|
||||
if replacement != original:
|
||||
connection.execute('UPDATE "' + table + '" SET "' + column + '"=? WHERE rowid=?', (replacement, rowid))
|
||||
connection.commit()
|
||||
connection.execute('VACUUM')
|
||||
if connection.execute('PRAGMA integrity_check').fetchone()[0] != 'ok':
|
||||
raise RuntimeError('原生分析历史数据库校验失败。')
|
||||
databases += 1
|
||||
finally:
|
||||
connection.close()
|
||||
return databases
|
||||
|
||||
|
||||
def rollback_plan(plan):
|
||||
# Only paths generated by this migration are eligible. Databases and source
|
||||
# archive trees are retained even after an interrupted restore.
|
||||
attempt = plan['attempt']
|
||||
if not re.fullmatch(r'[a-f0-9]{16}', attempt):
|
||||
raise RuntimeError('恢复日志无效。')
|
||||
for item in plan['privateSwitches']:
|
||||
target, previous, staged = (checked_local(item[key]) for key in ('target', 'previous', 'staged'))
|
||||
failed = checked_local(item['failed'])
|
||||
if previous.exists():
|
||||
if target.exists():
|
||||
target.rename(failed)
|
||||
previous.rename(target)
|
||||
elif not item['existed'] and not staged.exists() and target.exists():
|
||||
target.rename(failed)
|
||||
control.atomic(control.CONFIG, plan['previousConfig'])
|
||||
JOURNAL.rename(control.PRIVATE / ('migration-rolled-back-' + attempt + '.json'))
|
||||
|
||||
|
||||
def commit_plan(plan, receipt):
|
||||
control.atomic(JOURNAL, plan)
|
||||
try:
|
||||
for item in plan['privateSwitches']:
|
||||
target, previous, staged = (checked_local(item[key]) for key in ('target', 'previous', 'staged'))
|
||||
if target.is_symlink() or staged.is_symlink() or previous.exists():
|
||||
raise RuntimeError('私有配置切换路径无效。')
|
||||
if target.exists():
|
||||
target.rename(previous)
|
||||
staged.rename(target)
|
||||
control.atomic(control.CONFIG, plan['newConfig'])
|
||||
control.atomic(RECEIPT, receipt)
|
||||
except BaseException:
|
||||
rollback_plan(plan)
|
||||
raise
|
||||
JOURNAL.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def restore(check_runtime=True):
|
||||
control.private_directory(control.PRIVATE)
|
||||
control.private_directory(control.DATA)
|
||||
(control.DATA / 'logs').mkdir(exist_ok=True)
|
||||
(control.DATA / 'sources').mkdir(exist_ok=True)
|
||||
(control.DATA / 'archive').mkdir(exist_ok=True)
|
||||
lock = os.open(control.PRIVATE / 'migration.lock', os.O_CREAT | os.O_RDWR, 0o600)
|
||||
try:
|
||||
control.host.fcntl.flock(lock, control.host.fcntl.LOCK_EX | control.host.fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
os.close(lock)
|
||||
raise RuntimeError('另一项迁移正在进行,请等待原窗口完成。') from None
|
||||
try:
|
||||
return restore_locked(check_runtime)
|
||||
finally:
|
||||
os.close(lock)
|
||||
|
||||
|
||||
def restore_locked(check_runtime):
|
||||
manifest_path = PAYLOAD / 'manifest.json'
|
||||
manifest = json.loads(manifest_path.read_text(encoding='utf-8'))
|
||||
if manifest.get('formatVersion') != 1 or not re.fullmatch(r'[a-f0-9]{32}', manifest.get('migrationId', '')):
|
||||
raise RuntimeError('迁移包格式无效。')
|
||||
migration_id = manifest['migrationId']
|
||||
if RECEIPT.exists():
|
||||
receipt = json.loads(RECEIPT.read_text(encoding='utf-8'))
|
||||
current = control.configuration()
|
||||
if receipt.get('migrationId') == migration_id and current.get('restoredFromMigration') == migration_id and current.get('mysqlDatabase') == receipt.get('database'):
|
||||
print('此迁移已完成,保留 Windows 上的后续数据,直接启动。', flush=True)
|
||||
JOURNAL.unlink(missing_ok=True)
|
||||
return receipt
|
||||
raise RuntimeError('此目录已有其他迁移记录,请使用新的部署目录。')
|
||||
stop_owned_service()
|
||||
if JOURNAL.exists():
|
||||
rollback_plan(json.loads(JOURNAL.read_text(encoding='utf-8')))
|
||||
print('上次未完成的切换已撤回,原配置已保留。', flush=True)
|
||||
value = control.verify() if check_runtime else control.configuration()
|
||||
print('正在校验迁移包……', flush=True)
|
||||
for name, key in (('database.sql.gz', 'databaseDumpSha256'), ('archive.tar.gz', 'archiveSha256'),
|
||||
('native-history.tar.gz', 'nativeHistorySha256')):
|
||||
if sha256(PAYLOAD / name) != manifest[key]:
|
||||
raise RuntimeError('迁移包文件不完整,已停止恢复。')
|
||||
if manifest.get('modelConfigSha256') and sha256(PAYLOAD / 'model-config.json') != manifest['modelConfigSha256']:
|
||||
raise RuntimeError('模型配置校验失败。')
|
||||
if shutil.disk_usage(ROOT).free < manifest['minimumFreeBytes']:
|
||||
raise RuntimeError('可用磁盘空间不足,请至少预留 ' + str((manifest['minimumFreeBytes'] + 1024**3 - 1) // 1024**3) + ' GB。')
|
||||
attempt = uuid.uuid4().hex[:16]
|
||||
stage = control.PRIVATE / ('migration-stage-' + attempt)
|
||||
stage.mkdir(mode=0o700)
|
||||
archive = control.DATA / 'migrations' / attempt / 'archive'
|
||||
archive.parent.mkdir(parents=True, mode=0o700)
|
||||
native = stage / 'nao-users'
|
||||
print('正在恢复来源文件和分析历史……', flush=True)
|
||||
extract_checked(PAYLOAD / 'archive.tar.gz', archive, manifest['archiveFiles'])
|
||||
extract_checked(PAYLOAD / 'native-history.tar.gz', native, manifest['nativeHistoryFiles'])
|
||||
native_count = relocate_native(native, manifest['sourceProjectRoot'])
|
||||
if native_count != manifest['nativeHistoryDatabases']:
|
||||
raise RuntimeError('分析历史文件数量不一致。')
|
||||
model = stage / 'model-config.json'
|
||||
if manifest.get('modelConfigSha256'):
|
||||
shutil.copy2(PAYLOAD / 'model-config.json', model)
|
||||
model.chmod(0o600)
|
||||
settings = json.loads(model.read_text(encoding='utf-8'))
|
||||
if any('mimo' in str(profile.get(key, '')).lower() for profile in settings['profiles'] for key in ('model', 'name', 'baseUrl')):
|
||||
raise RuntimeError('迁移模型配置中仍含 MiMo。')
|
||||
database = 'quality_restore_' + uuid.uuid4().hex
|
||||
try:
|
||||
start_database(value)
|
||||
run_query(value, 'CREATE DATABASE `' + database + '` CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;')
|
||||
restore_dump(value, database, PAYLOAD / 'database.sql.gz')
|
||||
query = lambda statement: run_query(value, statement, database)
|
||||
print('正在核对全部数据表、原账号和正式统计结果……', flush=True)
|
||||
actual_counts = table_counts(query)
|
||||
if actual_counts != manifest['tableCounts']:
|
||||
raise RuntimeError('恢复后的数据表行数不一致,未切换正式使用库。')
|
||||
actual_users = [json.loads(line) for line in query("SELECT JSON_OBJECT('id',id,'username',username,'passwordHashSha256',SHA2(password_hash,256),'active',active) FROM app_user ORDER BY id;").splitlines()]
|
||||
if sorted(actual_users, key=lambda row: row['id']) != sorted(manifest['sourceUsers'], key=lambda row: row['id']):
|
||||
raise RuntimeError('原用户账号校验失败,未切换使用库。')
|
||||
if query_evidence(query) != manifest['queryEvidence']:
|
||||
raise RuntimeError('正式版本和月度统计结果不一致,未切换使用库。')
|
||||
rebound = rebind_sources(query, manifest, archive)
|
||||
run_query(value, 'GRANT ALL PRIVILEGES ON `' + database + "`.* TO 'quality_app'@'127.0.0.1';")
|
||||
run_query(value, 'FLUSH TABLES;')
|
||||
finally:
|
||||
stop_owned_service()
|
||||
new_config = {**value, 'mysqlDatabase': database, 'archiveDirectory': archive.relative_to(ROOT).as_posix(),
|
||||
'restoredFromMigration': migration_id, 'webBind': '0.0.0.0'}
|
||||
for name in ('dataWorkflowTestMode', 'analysisAutonomousEnabled', 'analysisProtocolV1Enabled'):
|
||||
if name in manifest.get('applicationSettings', {}):
|
||||
new_config[name] = bool(manifest['applicationSettings'][name])
|
||||
switches = []
|
||||
for staged, target in [(native, control.PRIVATE / 'nao-users')] + ([(model, control.PRIVATE / 'nao-model-config.json')] if model.exists() else []):
|
||||
switches.append({'staged': staged.relative_to(ROOT).as_posix(), 'target': target.relative_to(ROOT).as_posix(),
|
||||
'previous': (control.PRIVATE / ('before-migration-' + attempt + '-' + target.name)).relative_to(ROOT).as_posix(),
|
||||
'failed': (control.PRIVATE / ('failed-migration-' + attempt + '-' + target.name)).relative_to(ROOT).as_posix(),
|
||||
'existed': target.exists()})
|
||||
receipt = {'migrationId': migration_id, 'migrationManifestSha256': sha256(manifest_path), 'database': database,
|
||||
'restoredAtUtc': datetime.now(timezone.utc).isoformat(), 'tables': len(actual_counts),
|
||||
'totalRows': sum(actual_counts.values()), 'sourceReferencesRebound': rebound,
|
||||
'modelProfilesMigrated': manifest['modelProfilesMigrated'], 'modelProfilesExcluded': manifest['modelProfilesExcluded'],
|
||||
'nativeHistoryDatabases': native_count, 'originalDatabasesPreserved': True,
|
||||
'archiveDirectory': new_config['archiveDirectory']}
|
||||
plan = {'attempt': attempt, 'previousConfig': value, 'newConfig': new_config, 'privateSwitches': switches}
|
||||
commit_plan(plan, receipt)
|
||||
print('迁移恢复完成。请使用原来的账号和密码登录;MiMo 配置未迁移。', flush=True)
|
||||
return receipt
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
restore()
|
||||
except (RuntimeError, OSError, ValueError, subprocess.SubprocessError) as error:
|
||||
import traceback
|
||||
log = control.DATA / 'logs/migration-validation.log'
|
||||
log.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd = os.open(log, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o600)
|
||||
with os.fdopen(fd, 'w', encoding='utf-8') as output:
|
||||
traceback.print_exc(file=output)
|
||||
if isinstance(error, RuntimeError):
|
||||
print(str(error), file=sys.stderr)
|
||||
print('迁移未完成。已有数据库保留;原配置在切换失败时自动恢复。请查看私有恢复日志后重试。', file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,87 @@
|
||||
"""Platform helpers for the standalone distribution; POSIX behavior is preserved."""
|
||||
from __future__ import annotations
|
||||
|
||||
import errno
|
||||
import os
|
||||
import signal
|
||||
import time
|
||||
IS_WINDOWS = os.name == 'nt'
|
||||
|
||||
if os.name != 'nt':
|
||||
import fcntl
|
||||
else:
|
||||
import msvcrt
|
||||
|
||||
class _FileLocks:
|
||||
LOCK_EX, LOCK_NB, LOCK_UN = 2, 4, 8
|
||||
|
||||
@staticmethod
|
||||
def flock(fd, operation):
|
||||
os.lseek(fd, 0, os.SEEK_SET)
|
||||
if operation & _FileLocks.LOCK_UN:
|
||||
msvcrt.locking(fd, msvcrt.LK_UNLCK, 1)
|
||||
return
|
||||
while True:
|
||||
try:
|
||||
msvcrt.locking(fd, msvcrt.LK_NBLCK, 1)
|
||||
return
|
||||
except OSError as error:
|
||||
if error.errno not in (errno.EACCES, errno.EAGAIN, errno.EDEADLK):
|
||||
raise
|
||||
if operation & _FileLocks.LOCK_NB:
|
||||
raise BlockingIOError(errno.EAGAIN, 'File is locked') from None
|
||||
time.sleep(.05)
|
||||
|
||||
fcntl = _FileLocks()
|
||||
|
||||
|
||||
def private_bits(mode):
|
||||
# NTFS access is protected by the inherited user-only ACL on .runtime.
|
||||
# Windows' st_mode has no information about that ACL.
|
||||
return not IS_WINDOWS and bool(mode & 0o077)
|
||||
|
||||
|
||||
def owner_mismatch(info):
|
||||
return not IS_WINDOWS and info.st_uid != os.getuid()
|
||||
|
||||
|
||||
def fchmod(fd, mode):
|
||||
if os.name != 'nt':
|
||||
os.fchmod(fd, mode)
|
||||
|
||||
|
||||
def ps_value(pid, field):
|
||||
import psutil
|
||||
try:
|
||||
process = psutil.Process(int(pid))
|
||||
if not process.is_running() or process.status() == psutil.STATUS_ZOMBIE:
|
||||
return None
|
||||
if field == 'lstart':
|
||||
return repr(process.create_time())
|
||||
if field == 'command':
|
||||
return ' '.join(process.cmdline())
|
||||
return None
|
||||
except (psutil.Error, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def group_leader(pid):
|
||||
return True if os.name == 'nt' else os.getpgid(pid) == pid
|
||||
|
||||
|
||||
def killpg(pid, sig):
|
||||
if os.name != 'nt':
|
||||
return os.killpg(pid, sig)
|
||||
import psutil
|
||||
try:
|
||||
process = psutil.Process(pid)
|
||||
# Caller verifies creation time and executable before reaching this point.
|
||||
children = process.children(recursive=True)
|
||||
for child in reversed(children):
|
||||
try:
|
||||
child.kill() if sig == getattr(signal, 'SIGKILL', 9) else child.terminate()
|
||||
except psutil.NoSuchProcess:
|
||||
pass
|
||||
process.kill() if sig == getattr(signal, 'SIGKILL', 9) else process.terminate()
|
||||
except psutil.NoSuchProcess:
|
||||
raise ProcessLookupError(pid) from None
|
||||
@@ -0,0 +1,117 @@
|
||||
"""Exercise an isolated empty MySQL database with the host's native test runtimes."""
|
||||
import argparse
|
||||
import http.cookiejar
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
from build_bundle import ROOT, TEMPLATES, adapt_runtime
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--work', type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
work = args.work.resolve()
|
||||
stage = Path(tempfile.mkdtemp(prefix='qc-offline-', dir='/tmp')).resolve()
|
||||
(work / 'smoke-root.txt').write_text(str(stage))
|
||||
for folder in ('deploy/offline', 'scripts', 'runtimes', 'app', '.runtime'):
|
||||
(stage / folder).mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
for name in ('control.py', 'gateway.py'):
|
||||
shutil.copy2(TEMPLATES / name, stage / 'deploy/offline' / name)
|
||||
shutil.copy2(TEMPLATES / 'runtime_platform.py', stage / 'scripts/runtime_platform.py')
|
||||
for pattern in ('nao_*.py', 'analysis_visual*.py'):
|
||||
for path in (ROOT / 'scripts').glob(pattern):
|
||||
(stage / 'scripts' / path.name).write_text(adapt_runtime(path.read_text()) if path.name == 'nao_quality_runtime.py' else path.read_text())
|
||||
shutil.copytree(ROOT / 'ai/nao-quality', stage / 'ai/nao-quality')
|
||||
shutil.copy2(next((work / 'source/backend/target').glob('*.jar')), stage / 'app/backend.jar')
|
||||
shutil.copytree(work / 'source/web/dist', stage / 'app/web')
|
||||
for name, target in (('java', ROOT / '.runtime/jdk/Contents/Home'), ('mysql', ROOT / '.runtime/mysql'),
|
||||
('python', ROOT / '.runtime/nao-trial/venv')):
|
||||
(stage / 'runtimes' / name).symlink_to(target, target_is_directory=True)
|
||||
def available(port):
|
||||
try:
|
||||
with socket.socket() as test:
|
||||
test.bind(('127.0.0.1', port))
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
port = next(p for p in (5284, 5283, 5285) if available(p))
|
||||
def free():
|
||||
with socket.socket() as test:
|
||||
test.bind(('127.0.0.1', 0))
|
||||
return test.getsockname()[1]
|
||||
value = {'mysqlPort': free(), 'backendPort': port, 'webPort': free(), 'webBind': '127.0.0.1',
|
||||
'mysqlPassword': secrets.token_urlsafe(32), 'rootPassword': secrets.token_urlsafe(32),
|
||||
'adminUsername': 'admin', 'adminPassword': secrets.token_urlsafe(24)}
|
||||
config = stage / '.runtime/deployment.json'
|
||||
config.write_text(json.dumps(value))
|
||||
config.chmod(0o600)
|
||||
report = {'environment': 'macOS isolated empty database using host-native runtimes',
|
||||
'targetWindowsLinuxNativeValidated': False, 'checks': []}
|
||||
process = None
|
||||
try:
|
||||
for cycle in range(2):
|
||||
with (stage / f'cycle-{cycle}.log').open('w') as log:
|
||||
process = subprocess.Popen([sys.executable, str(stage / 'deploy/offline/control.py'), 'serve'], stdout=log, stderr=subprocess.STDOUT)
|
||||
for _ in range(180):
|
||||
if process.poll() is not None:
|
||||
raise RuntimeError('Supervisor exited; inspect ' + str(stage / f'cycle-{cycle}.log'))
|
||||
try:
|
||||
state = json.loads((stage / '.runtime/service.json').read_text())
|
||||
if state.get('ready'):
|
||||
break
|
||||
except (FileNotFoundError, ValueError):
|
||||
pass
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise RuntimeError('Isolated startup timed out')
|
||||
cookies = http.cookiejar.CookieJar()
|
||||
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}), urllib.request.HTTPCookieProcessor(cookies))
|
||||
base = f'http://127.0.0.1:{value["webPort"]}'
|
||||
with opener.open(base + '/', timeout=5) as response:
|
||||
assert response.status == 200 and b'<div id="root">' in response.read()
|
||||
with opener.open(base + '/actuator/health', timeout=5) as response:
|
||||
assert json.load(response)['status'] == 'UP'
|
||||
with opener.open(base + '/api/auth/csrf', timeout=5) as response:
|
||||
token = json.load(response)['token']
|
||||
request = urllib.request.Request(base + '/api/auth/login',
|
||||
data=json.dumps({'username': value['adminUsername'], 'password': value['adminPassword']}).encode(),
|
||||
headers={'Content-Type': 'application/json', 'X-XSRF-TOKEN': token})
|
||||
with opener.open(request, timeout=10) as response:
|
||||
assert response.status == 200
|
||||
for path in ('/api/auth/me', '/api/options'):
|
||||
with opener.open(base + path, timeout=10) as response:
|
||||
assert response.status == 200
|
||||
report['checks'].append({'cycle': cycle + 1, 'web': True, 'databaseMigration': True,
|
||||
'csrfAndLogin': True, 'authenticatedOptions': True})
|
||||
result = subprocess.run([sys.executable, str(stage / 'deploy/offline/control.py'), 'stop'], capture_output=True, text=True, timeout=150)
|
||||
if result.returncode:
|
||||
raise RuntimeError('Stop failed: ' + result.stderr)
|
||||
process.wait(timeout=10)
|
||||
process = None
|
||||
assert not (stage / '.runtime/service.json').exists()
|
||||
print('Isolated startup/login/stop cycle', cycle + 1, 'passed', flush=True)
|
||||
report['passed'] = True
|
||||
except Exception as error:
|
||||
report['passed'] = False
|
||||
report['error'] = str(error)
|
||||
print(str(error), flush=True)
|
||||
finally:
|
||||
if process is not None and process.poll() is None:
|
||||
subprocess.run([sys.executable, str(stage / 'deploy/offline/control.py'), 'stop'], capture_output=True, timeout=150)
|
||||
(work / 'smoke-report.json').write_text(json.dumps(report, ensure_ascii=False, indent=2) + '\n')
|
||||
print(json.dumps(report, ensure_ascii=False), flush=True)
|
||||
return 0 if report['passed'] else 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,148 @@
|
||||
import gzip
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sqlite3
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(HERE))
|
||||
import control
|
||||
import restore_migration as restore
|
||||
from migration_common import dump_counts, extract_checked, filter_models, inserted_rows, path_under_source, relative_name
|
||||
from prepare_migration import native_copy
|
||||
|
||||
|
||||
class MigrationValidationTest(unittest.TestCase):
|
||||
def test_extended_insert_counts_ignore_strings_and_escaped_quotes(self):
|
||||
cases = [
|
||||
(b"INSERT INTO `x` VALUES (1,'a'),(2,'b');\n", 2),
|
||||
(b"INSERT INTO `x` VALUES (1,'fake ),( delimiter'),(2,'quote \\' ),( escaped'),(3,0x001122);\n", 3),
|
||||
(b"INSERT INTO `x` VALUES ('a\\\\',NULL,'(nested)'),(NULL,'\\n',4);\n", 2),
|
||||
(b"INSERT INTO `x` VALUES ('\xe4\xb8\xad\xe6\x96\x87', '{\\\"text\\\":\\\"),(\\\"}');\n", 1),
|
||||
]
|
||||
for statement, count in cases:
|
||||
with self.subTest(statement=statement):
|
||||
self.assertEqual(inserted_rows(statement), count)
|
||||
|
||||
def test_dump_counts_include_empty_tables(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
path = Path(folder) / 'dump.gz'
|
||||
with gzip.open(path, 'wb') as stream:
|
||||
stream.write(b"CREATE TABLE `empty` (id INT);\nCREATE TABLE `x` (id INT);\nINSERT INTO `x` VALUES (1),(2);\n")
|
||||
counts, size = dump_counts(path)
|
||||
self.assertEqual(counts, {'empty': 0, 'x': 2})
|
||||
self.assertGreater(size, 0)
|
||||
|
||||
def test_mimo_profiles_and_root_selection_are_excluded(self):
|
||||
config = {'revision': 'original', 'activeProfileId': 'mimo', 'profiles': [
|
||||
{'id': 'mimo', 'name': 'MiMo', 'model': 'mimo-v2.6', 'apiKey': 'exclude-this', 'baseUrl': 'https://api.example.test'},
|
||||
{'id': 'other', 'name': 'Qwen', 'model': 'qwen-test', 'apiKey': 'retain-this', 'baseUrl': 'https://other.example.test'}]}
|
||||
filtered, removed = filter_models(config)
|
||||
self.assertEqual(len(removed), 1)
|
||||
self.assertEqual(filtered['activeProfileId'], 'other')
|
||||
self.assertEqual(filtered['apiKey'], 'retain-this')
|
||||
self.assertEqual(filtered['revision'], 'original')
|
||||
self.assertNotIn('exclude-this', json.dumps(filtered))
|
||||
self.assertEqual(len(filtered['profiles']), 1)
|
||||
|
||||
def test_mac_path_is_parsed_portably(self):
|
||||
self.assertEqual(path_under_source('/Users/a/archive/source/文件.xlsx', '/Users/a/archive'), 'source/文件.xlsx')
|
||||
for name in ('../outside', '/absolute', 'C:/data', 'a\\b', '.', 'a/./b', 'CON.txt', 'x/name.'):
|
||||
with self.subTest(name=name), self.assertRaises(ValueError):
|
||||
relative_name(name)
|
||||
with self.assertRaises(ValueError):
|
||||
path_under_source('/Users/a/archive-other/secret', '/Users/a/archive')
|
||||
|
||||
def test_archive_rejects_traversal_links_duplicates_and_bad_hash(self):
|
||||
for kind in ('traversal', 'link', 'duplicate', 'checksum'):
|
||||
with self.subTest(kind=kind), tempfile.TemporaryDirectory() as directory:
|
||||
folder = Path(directory)
|
||||
archive = folder / 'archive.gz'
|
||||
body = b'contents'
|
||||
with tarfile.open(archive, 'w:gz') as output:
|
||||
info = tarfile.TarInfo('../outside' if kind == 'traversal' else 'file')
|
||||
if kind == 'link':
|
||||
info.type, info.linkname = tarfile.SYMTYPE, '/outside'
|
||||
output.addfile(info)
|
||||
else:
|
||||
info.size = len(body)
|
||||
output.addfile(info, io.BytesIO(body))
|
||||
if kind == 'duplicate':
|
||||
output.addfile(info, io.BytesIO(body))
|
||||
expected = {'file': {'bytes': len(body), 'sha256': '0' * 64 if kind == 'checksum' else hashlib.sha256(body).hexdigest()}}
|
||||
with self.assertRaises(ValueError):
|
||||
extract_checked(archive, folder / 'target', expected)
|
||||
self.assertFalse((folder / 'outside').exists())
|
||||
|
||||
def test_native_history_preserves_chats_and_scrubs_credentials(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
folder = Path(directory)
|
||||
source = folder / 'source.sqlite'
|
||||
db = sqlite3.connect(source)
|
||||
db.executescript("CREATE TABLE chat(id TEXT,body TEXT);INSERT INTO chat VALUES('a','history');"
|
||||
"CREATE TABLE project_llm_config(api_key TEXT);INSERT INTO project_llm_config VALUES('mimo-exclusive-secret');"
|
||||
"CREATE TABLE session(token TEXT);INSERT INTO session VALUES('transient-token');")
|
||||
db.commit()
|
||||
db.close()
|
||||
target = folder / 'target.sqlite'
|
||||
native_copy(source, target)
|
||||
migrated = sqlite3.connect(target)
|
||||
self.assertEqual(migrated.execute('SELECT body FROM chat').fetchone()[0], 'history')
|
||||
self.assertEqual(migrated.execute('SELECT COUNT(*) FROM project_llm_config').fetchone()[0], 0)
|
||||
self.assertEqual(migrated.execute('SELECT COUNT(*) FROM session').fetchone()[0], 0)
|
||||
migrated.close()
|
||||
self.assertNotIn(b'mimo-exclusive-secret', target.read_bytes())
|
||||
|
||||
|
||||
class MigrationCommitTest(unittest.TestCase):
|
||||
def test_partial_file_switch_rolls_back_existing_files_and_configuration(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
private = root / '.runtime'
|
||||
private.mkdir()
|
||||
target = private / 'nao-users'
|
||||
target.mkdir()
|
||||
(target / 'old').write_text('original')
|
||||
staged = private / 'stage-native'
|
||||
staged.mkdir()
|
||||
(staged / 'new').write_text('new')
|
||||
model = private / 'nao-model-config.json'
|
||||
model.write_text('original model')
|
||||
attempt = 'a' * 16
|
||||
plan = {'attempt': attempt, 'previousConfig': {'previous': True}, 'newConfig': {'new': True}, 'privateSwitches': [
|
||||
{'target': '.runtime/nao-users', 'previous': '.runtime/previous-native', 'staged': '.runtime/stage-native', 'failed': '.runtime/failed-native', 'existed': True},
|
||||
{'target': '.runtime/nao-model-config.json', 'previous': '.runtime/previous-model', 'staged': '.runtime/missing-stage-model', 'failed': '.runtime/failed-model', 'existed': True}]}
|
||||
with patch.multiple(restore, ROOT=root, JOURNAL=private / 'journal', RECEIPT=private / 'receipt'), patch.object(control, 'CONFIG', private / 'deployment.json'):
|
||||
with self.assertRaises(FileNotFoundError):
|
||||
restore.commit_plan(plan, {})
|
||||
self.assertEqual((target / 'old').read_text(), 'original')
|
||||
self.assertEqual(model.read_text(), 'original model')
|
||||
self.assertEqual(json.loads((private / 'deployment.json').read_text()), {'previous': True})
|
||||
self.assertTrue((private / 'failed-native/new').exists())
|
||||
|
||||
def test_repeated_restore_does_not_reimport_old_snapshot(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
payload = root / 'migration'
|
||||
payload.mkdir()
|
||||
migration_id = 'a' * 32
|
||||
(payload / 'manifest.json').write_text(json.dumps({'formatVersion': 1, 'migrationId': migration_id}))
|
||||
receipt = root / 'receipt'
|
||||
value = {'migrationId': migration_id, 'database': 'quality_restore_' + 'b' * 32}
|
||||
receipt.write_text(json.dumps(value))
|
||||
config = {'restoredFromMigration': migration_id, 'mysqlDatabase': value['database']}
|
||||
with patch.multiple(restore, PAYLOAD=payload, RECEIPT=receipt, JOURNAL=root / 'journal'), patch.object(control, 'configuration', return_value=config), patch.object(restore, 'stop_owned_service') as stop, patch.object(restore, 'run_query') as query:
|
||||
self.assertEqual(restore.restore_locked(False), value)
|
||||
stop.assert_not_called()
|
||||
query.assert_not_called()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,179 @@
|
||||
import functools
|
||||
import http.client
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(HERE))
|
||||
from gateway import make_server
|
||||
from build_bundle import adapt_runtime
|
||||
import runtime_platform
|
||||
|
||||
|
||||
class Upstream(BaseHTTPRequestHandler):
|
||||
def log_message(self, *_):
|
||||
pass
|
||||
|
||||
def do_GET(self):
|
||||
if self.path == '/api/events':
|
||||
self.send_response(200)
|
||||
self.send_header('Content-Type', 'text/event-stream')
|
||||
self.end_headers()
|
||||
self.wfile.write(b'data: first\n\n')
|
||||
self.wfile.flush()
|
||||
time.sleep(.8)
|
||||
self.wfile.write(b'data: second\n\n')
|
||||
self.wfile.flush()
|
||||
else:
|
||||
body = json.dumps({'cookie': self.headers.get('Cookie'), 'host': self.headers.get('Host')}).encode()
|
||||
self.send_response(401)
|
||||
self.send_header('Set-Cookie', 'one=1; HttpOnly')
|
||||
self.send_header('Set-Cookie', 'two=2; SameSite=Strict')
|
||||
self.send_header('Content-Length', str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def do_POST(self):
|
||||
body = self.rfile.read(int(self.headers['Content-Length']))
|
||||
self.send_response(201)
|
||||
self.send_header('Content-Length', str(len(body)))
|
||||
self.send_header('X-CSRF-Echo', self.headers.get('X-XSRF-TOKEN', ''))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
|
||||
class GatewayTest(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.temp = tempfile.TemporaryDirectory()
|
||||
root = Path(cls.temp.name)
|
||||
(root / 'index.html').write_text('QUALITY APP', encoding='utf-8')
|
||||
(root / 'assets').mkdir()
|
||||
(root / 'assets/file.js').write_text('const x=1;')
|
||||
cls.backend = ThreadingHTTPServer(('127.0.0.1', 0), Upstream)
|
||||
cls.frontend = make_server(root, '127.0.0.1', 0, cls.backend.server_port)
|
||||
for server in (cls.backend, cls.frontend):
|
||||
threading.Thread(target=server.serve_forever, daemon=True).start()
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
for server in (cls.frontend, cls.backend):
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
cls.temp.cleanup()
|
||||
|
||||
def connection(self):
|
||||
return http.client.HTTPConnection('127.0.0.1', self.frontend.server_port, timeout=3)
|
||||
|
||||
def test_static_and_spa(self):
|
||||
for url, body in [('/', b'QUALITY APP'), ('/?page=home', b'QUALITY APP'), ('/analysis', b'QUALITY APP'), ('/assets/file.js', b'const x=1;')]:
|
||||
connection = self.connection()
|
||||
connection.request('GET', url)
|
||||
response = connection.getresponse()
|
||||
self.assertEqual(response.status, 200)
|
||||
self.assertEqual(response.read(), body)
|
||||
connection.close()
|
||||
|
||||
def test_no_directory_listing_or_private_files(self):
|
||||
for url in ('/assets/', '/.runtime/deployment.json', '/../.runtime/deployment.json', '/missing.js'):
|
||||
connection = self.connection()
|
||||
connection.request('GET', url)
|
||||
response = connection.getresponse()
|
||||
self.assertEqual(response.status, 404)
|
||||
response.read()
|
||||
connection.close()
|
||||
|
||||
def test_auth_cookies_and_host_are_preserved(self):
|
||||
connection = self.connection()
|
||||
connection.request('GET', '/api/auth/me', headers={'Cookie': 'SESSION=abc', 'Host': '192.168.1.22:5280'})
|
||||
response = connection.getresponse()
|
||||
self.assertEqual(response.status, 401)
|
||||
self.assertEqual(len(response.headers.get_all('Set-Cookie')), 2)
|
||||
self.assertEqual(json.loads(response.read()), {'cookie': 'SESSION=abc', 'host': '192.168.1.22:5280'})
|
||||
connection.close()
|
||||
|
||||
def test_upload_and_csrf(self):
|
||||
connection = self.connection()
|
||||
data = os.urandom(2 * 1024 * 1024 + 19)
|
||||
connection.request('POST', '/api/sources/upload', data, {'X-XSRF-TOKEN': 'csrf-value'})
|
||||
response = connection.getresponse()
|
||||
self.assertEqual(response.status, 201)
|
||||
self.assertEqual(response.getheader('X-CSRF-Echo'), 'csrf-value')
|
||||
self.assertEqual(response.read(), data)
|
||||
connection.close()
|
||||
|
||||
def test_sse_is_streamed_before_upstream_finishes(self):
|
||||
connection = self.connection()
|
||||
started = time.monotonic()
|
||||
connection.request('GET', '/api/events')
|
||||
response = connection.getresponse()
|
||||
self.assertEqual(response.readline(), b'data: first\n')
|
||||
self.assertLess(time.monotonic() - started, .65)
|
||||
self.assertIn(b'data: second', response.read())
|
||||
connection.close()
|
||||
|
||||
|
||||
class PortabilityTest(unittest.TestCase):
|
||||
def test_port_probe_rejects_live_listener_and_allows_restart(self):
|
||||
import control
|
||||
with socket.socket() as listener:
|
||||
listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
listener.bind(('127.0.0.1', 0))
|
||||
port = listener.getsockname()[1]
|
||||
listener.listen()
|
||||
with self.assertRaises(RuntimeError):
|
||||
control.free_port(port)
|
||||
accepted, _ = listener.accept()
|
||||
accepted.close()
|
||||
control.free_port(port)
|
||||
|
||||
def test_process_identity_requires_start_time_executable_and_arguments(self):
|
||||
import psutil
|
||||
import control
|
||||
process = psutil.Process()
|
||||
info = control.record(process, process.cmdline())
|
||||
self.assertTrue(control.matches(info))
|
||||
self.assertFalse(control.matches({**info, 'created': info['created'] - 10}))
|
||||
self.assertFalse(control.matches({**info, 'executable': '/unrelated/program'}))
|
||||
self.assertFalse(control.matches({**info, 'args': ['unrelated-command']}))
|
||||
|
||||
def test_posix_permissions_are_not_relaxed(self):
|
||||
if os.name != 'nt':
|
||||
self.assertFalse(runtime_platform.private_bits(0o100600))
|
||||
self.assertTrue(runtime_platform.private_bits(0o100640))
|
||||
|
||||
def test_lock_excludes_another_process(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory) / 'lock'
|
||||
fd = os.open(path, os.O_CREAT | os.O_RDWR, 0o600)
|
||||
runtime_platform.fcntl.flock(fd, runtime_platform.fcntl.LOCK_EX | runtime_platform.fcntl.LOCK_NB)
|
||||
code = ('import sys,os;sys.path.insert(0,sys.argv[1]);import runtime_platform as h;'
|
||||
'fd=os.open(sys.argv[2],os.O_RDWR);h.fcntl.flock(fd,h.fcntl.LOCK_EX|h.fcntl.LOCK_NB)')
|
||||
result = subprocess.run([sys.executable, '-c', code, str(HERE), str(path)], capture_output=True)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
os.close(fd)
|
||||
|
||||
def test_source_adapter_preserves_mac_source(self):
|
||||
path = HERE.parents[1] / 'scripts/nao_quality_runtime.py'
|
||||
source = path.read_text()
|
||||
adapted = adapt_runtime(source)
|
||||
self.assertIn('import fcntl', source)
|
||||
self.assertIn('from runtime_platform import fcntl', adapted)
|
||||
self.assertIn('_host.private_bits(directory_info.st_mode)', adapted)
|
||||
self.assertIn('_host.killpg', adapted)
|
||||
compile(adapted, str(path), 'exec')
|
||||
self.assertEqual(path.read_text(), source)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,175 @@
|
||||
"""Offline tests for the signed Windows update publisher."""
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import sys
|
||||
import unittest
|
||||
import zipfile
|
||||
|
||||
from cryptography.hazmat.primitives import serialization
|
||||
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import publish_update
|
||||
import update_manager
|
||||
|
||||
|
||||
class PublishUpdateTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.base = Path(self.temp.name)
|
||||
self.root = self.base / 'windows-bundle'
|
||||
self.root.mkdir()
|
||||
self.output = self.base / 'published'
|
||||
self.key_path = self.base / 'release-private-key.pem'
|
||||
self.private = Ed25519PrivateKey.generate()
|
||||
self.key_path.write_bytes(self.private.private_bytes(
|
||||
serialization.Encoding.PEM,
|
||||
serialization.PrivateFormat.PKCS8,
|
||||
serialization.NoEncryption(),
|
||||
))
|
||||
public_key_path = self.root / update_manager.KEY
|
||||
public_key_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
public_key_path.write_bytes(self.private.public_key().public_bytes(
|
||||
serialization.Encoding.PEM,
|
||||
serialization.PublicFormat.SubjectPublicKeyInfo,
|
||||
))
|
||||
self.data = {
|
||||
'app/backend.jar': b'backend program',
|
||||
'app/web/index.html': b'<html>program</html>',
|
||||
'deploy/offline/control.py': b'control program',
|
||||
'deploy/offline/update_manager.py': b'update manager program',
|
||||
'deploy/offline/publish_update.py': b'publisher program',
|
||||
'deploy/offline/windows-start.ps1': b'start script',
|
||||
'deploy/offline/README.md': b'package instructions',
|
||||
'backend/src/main/resources/analysis-runtime-policy.json': b'{"enabled":false}',
|
||||
'scripts/runtime.py': b'helper program',
|
||||
'ai/nao-quality/app.py': b'ai program',
|
||||
'runtimes/python/python.exe': b'local runtime must not ship in an update',
|
||||
'migration/manifest.json': b'private migration data',
|
||||
'deploy/bootstrap/update-recovery.py': b'immutable bootstrap',
|
||||
'.runtime/deployment.json': b'{"mysqlPassword":"private"}',
|
||||
'runtimedata/archive/source.xlsx': b'user business data',
|
||||
update_manager.KEY: public_key_path.read_bytes(),
|
||||
}
|
||||
for name, value in self.data.items():
|
||||
target = self.root.joinpath(*name.split('/'))
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_bytes(value)
|
||||
self.manifest = {
|
||||
'platform': 'windows',
|
||||
'architecture': 'x86_64',
|
||||
'updateVersion': '2026.11.04.1',
|
||||
'updateSequence': 1,
|
||||
'runtimes': [{'name': 'python', 'version': '3.11.15'}],
|
||||
'pythonWheels': [{'file': 'cryptography.whl', 'sha256': 'a' * 64}],
|
||||
'files': [
|
||||
{'path': name, 'sha256': hashlib.sha256(value).hexdigest()}
|
||||
for name, value in self.data.items()
|
||||
],
|
||||
}
|
||||
(self.root / 'manifest.json').write_text(json.dumps(self.manifest), encoding='utf-8')
|
||||
|
||||
def build(self, **changes):
|
||||
args = {
|
||||
'root': self.root,
|
||||
'version': '2026.11.04.2',
|
||||
'sequence': 2,
|
||||
'private_key_path': self.key_path,
|
||||
'output_dir': self.output,
|
||||
}
|
||||
args.update(changes)
|
||||
return publish_update.publish(**args)
|
||||
|
||||
def test_release_is_signed_program_snapshot_with_runtime_fingerprint(self):
|
||||
artifacts = self.build()
|
||||
self.assertTrue(artifacts.archive.is_file())
|
||||
self.assertTrue(artifacts.latest.is_file())
|
||||
envelope = update_manager.read(artifacts.latest)
|
||||
release = update_manager.validate_release(envelope, self.root)
|
||||
raw = base64.b64decode(envelope['payload'], validate=True)
|
||||
self.private.public_key().verify(base64.b64decode(envelope['signature'], validate=True), raw)
|
||||
self.assertEqual('2026.11.04.2', release['version'])
|
||||
self.assertEqual(2, release['sequence'])
|
||||
self.assertEqual(update_manager.runtime_fingerprint(self.manifest), release['runtimeFingerprint'])
|
||||
self.assertNotIn('bundleManifest', release)
|
||||
included = {entry['path'] for entry in release['files']}
|
||||
self.assertEqual({name for name in self.data if update_manager.program_path(name)}, included)
|
||||
self.assertFalse(included & {
|
||||
'runtimes/python/python.exe',
|
||||
'migration/manifest.json',
|
||||
'deploy/bootstrap/update-recovery.py',
|
||||
'.runtime/deployment.json',
|
||||
'runtimedata/archive/source.xlsx',
|
||||
update_manager.KEY,
|
||||
})
|
||||
with zipfile.ZipFile(artifacts.archive) as archive:
|
||||
self.assertEqual(included, set(archive.namelist()))
|
||||
self.assertEqual(self.data['app/backend.jar'], archive.read('app/backend.jar'))
|
||||
self.assertEqual(hashlib.sha256(artifacts.archive.read_bytes()).hexdigest(), release['archive']['sha256'])
|
||||
|
||||
def test_rejects_wrong_key_stale_sequence_and_package_mutation(self):
|
||||
with self.assertRaisesRegex(publish_update.PublishError, '不匹配'):
|
||||
wrong_path = self.base / 'wrong-private-key.pem'
|
||||
wrong_path.write_bytes(Ed25519PrivateKey.generate().private_bytes(
|
||||
serialization.Encoding.PEM,
|
||||
serialization.PrivateFormat.PKCS8,
|
||||
serialization.NoEncryption(),
|
||||
))
|
||||
self.build(private_key_path=wrong_path)
|
||||
with self.assertRaisesRegex(publish_update.PublishError, '大于部署包当前序号'):
|
||||
self.build(sequence=1)
|
||||
(self.root / 'app/backend.jar').write_bytes(b'changed after manifest')
|
||||
with self.assertRaisesRegex(publish_update.PublishError, '摘要不一致'):
|
||||
self.build()
|
||||
self.assertFalse(self.output.exists())
|
||||
|
||||
def test_allows_only_explicit_exact_baseline_and_still_rejects_lower_sequence(self):
|
||||
with self.assertRaisesRegex(publish_update.PublishError, '大于部署包当前序号'):
|
||||
self.build(version='2026.11.04.1', sequence=1)
|
||||
with self.assertRaisesRegex(publish_update.PublishError, '完全一致'):
|
||||
self.build(version='2026.11.04.2', sequence=1, allow_baseline=True)
|
||||
baseline = self.build(version='2026.11.04.1', sequence=1, allow_baseline=True)
|
||||
release = update_manager.validate_release(update_manager.read(baseline.latest), self.root)
|
||||
self.assertEqual('2026.11.04.1', release['version'])
|
||||
self.assertEqual(1, release['sequence'])
|
||||
|
||||
self.manifest['updateVersion'] = '2026.11.04.1'
|
||||
self.manifest['updateSequence'] = 2
|
||||
(self.root / 'manifest.json').write_text(json.dumps(self.manifest), encoding='utf-8')
|
||||
self.output = self.base / 'lower-sequence-output'
|
||||
with self.assertRaisesRegex(publish_update.PublishError, '不能低于'):
|
||||
self.build(version='2026.11.04.1', sequence=1, allow_baseline=True)
|
||||
|
||||
def test_rejects_sensitive_program_path_and_missing_program_inventory(self):
|
||||
sensitive_name = 'ai/nao-quality/private/model.json'
|
||||
sensitive_path = self.root.joinpath(*sensitive_name.split('/'))
|
||||
sensitive_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
sensitive_path.write_bytes(b'private model config')
|
||||
self.manifest['files'].append({'path': sensitive_name, 'sha256': hashlib.sha256(sensitive_path.read_bytes()).hexdigest()})
|
||||
(self.root / 'manifest.json').write_text(json.dumps(self.manifest), encoding='utf-8')
|
||||
with self.assertRaisesRegex(publish_update.PublishError, '受保护'):
|
||||
self.build()
|
||||
self.manifest['files'].pop()
|
||||
self.manifest['files'] = [entry for entry in self.manifest['files'] if entry['path'] != 'scripts/runtime.py']
|
||||
(self.root / 'manifest.json').write_text(json.dumps(self.manifest), encoding='utf-8')
|
||||
with self.assertRaisesRegex(publish_update.PublishError, '完整程序文件清单'):
|
||||
self.build()
|
||||
|
||||
def test_refuses_key_inside_source_or_output_and_refuses_overwrite(self):
|
||||
with self.assertRaisesRegex(publish_update.PublishError, '部署包和更新产物目录之外'):
|
||||
self.build(private_key_path=self.root / 'manifest.json')
|
||||
with self.assertRaisesRegex(publish_update.PublishError, '部署包目录之外'):
|
||||
self.build(output_dir=self.root / 'updates')
|
||||
self.build()
|
||||
original_latest = (self.output / 'latest.json').read_bytes()
|
||||
with self.assertRaisesRegex(publish_update.PublishError, '同名发布文件'):
|
||||
self.build()
|
||||
self.assertEqual(original_latest, (self.output / 'latest.json').read_bytes())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,339 @@
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import textwrap
|
||||
import time
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
import psutil
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import update_bootstrap as bootstrap
|
||||
|
||||
|
||||
class ImmutableRecoveryTest(unittest.TestCase):
|
||||
def fixture(self, folder):
|
||||
root = Path(folder)
|
||||
directory = root / '.runtime/update'
|
||||
backup = directory / 'backup-20261104T120000Z-abcdef12'
|
||||
(backup / 'program/deploy/offline').mkdir(parents=True)
|
||||
(root / 'deploy/offline').mkdir(parents=True)
|
||||
old = {
|
||||
'app/backend.jar': b'verified old backend jar',
|
||||
'deploy/offline/control.py': b'VALID_OLD_VERSION = True\n',
|
||||
'deploy/offline/update_manager.py': b'def recover_locked(root):\n (root / ".runtime/restored").write_text("done")\n return {"recovered": True}\n',
|
||||
}
|
||||
entries = []
|
||||
for name, content in old.items():
|
||||
path = backup / 'program' / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(content)
|
||||
(root / name).parent.mkdir(parents=True, exist_ok=True)
|
||||
(root / name).write_bytes(b'def broken syntax\n')
|
||||
entries.append({'path': name, 'sha256': hashlib.sha256(content).hexdigest()})
|
||||
(backup / 'manifest.json').write_text(json.dumps({'files': entries}))
|
||||
(backup / 'program.json').write_text(json.dumps({'all': list(old), 'present': list(old)}))
|
||||
journal = {'phase': 'installing', 'databaseBackedUp': True, 'backup': backup.name,
|
||||
'backupManifestSha256': bootstrap.digest(backup / 'manifest.json'),
|
||||
'programInventorySha256': bootstrap.digest(backup / 'program.json')}
|
||||
(directory / 'journal.json').write_text(json.dumps(journal))
|
||||
return root, directory, backup, journal, old
|
||||
|
||||
def runtime_aliases(self, root):
|
||||
for role in ('supervisor', 'backend'):
|
||||
executable = bootstrap.runtime_executable(root, role)
|
||||
executable.parent.mkdir(parents=True, exist_ok=True)
|
||||
executable.symlink_to(psutil.Process().exe())
|
||||
|
||||
def record(self, process, name=None):
|
||||
observed = psutil.Process(process.pid)
|
||||
value = {'pid': process.pid, 'created': observed.create_time(),
|
||||
'executable': str(Path(observed.exe()).resolve()), 'args': observed.cmdline()[1:]}
|
||||
if name:
|
||||
value['name'] = name
|
||||
return value
|
||||
|
||||
def stop_test_child(self, process):
|
||||
if process.poll() is not None:
|
||||
return
|
||||
descendants = []
|
||||
try:
|
||||
descendants = psutil.Process(process.pid).children(recursive=True)
|
||||
except psutil.Error:
|
||||
pass
|
||||
process.terminate()
|
||||
try:
|
||||
process.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
process.kill()
|
||||
process.wait(timeout=5)
|
||||
for child in descendants:
|
||||
try:
|
||||
if child.is_running() and child.status() != psutil.STATUS_ZOMBIE:
|
||||
child.kill()
|
||||
except psutil.Error:
|
||||
pass
|
||||
|
||||
def wait_file(self, path, process):
|
||||
deadline = time.monotonic() + 10
|
||||
while time.monotonic() < deadline:
|
||||
if path.exists():
|
||||
return
|
||||
self.assertIsNone(process.poll(), 'Owned test child exited before readiness')
|
||||
time.sleep(.02)
|
||||
self.fail('Owned test child did not become ready')
|
||||
|
||||
def backend_script(self, root):
|
||||
path = root / 'backend-test.py'
|
||||
path.write_text(textwrap.dedent('''
|
||||
import hashlib, os, signal, sys, time
|
||||
from pathlib import Path
|
||||
root = Path(__file__).parent
|
||||
jar = root / 'app/backend.jar'
|
||||
held = jar.open('rb')
|
||||
def stop(*ignored):
|
||||
(root / '.runtime/backend-stopped-before-copy').write_text(hashlib.sha256(jar.read_bytes()).hexdigest())
|
||||
held.close()
|
||||
raise SystemExit(0)
|
||||
signal.signal(signal.SIGTERM, stop)
|
||||
(root / '.runtime/backend-ready').write_text(str(os.getpid()))
|
||||
while True:
|
||||
time.sleep(.02)
|
||||
'''))
|
||||
return path
|
||||
|
||||
def start_backend(self, root):
|
||||
self.runtime_aliases(root)
|
||||
script = self.backend_script(root)
|
||||
process = subprocess.Popen([str(bootstrap.runtime_executable(root, 'backend')), str(script), '-jar', str(root / 'app/backend.jar')],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
self.addCleanup(self.stop_test_child, process)
|
||||
self.wait_file(root / '.runtime/backend-ready', process)
|
||||
return process
|
||||
|
||||
def start_supervisor(self, root):
|
||||
self.runtime_aliases(root)
|
||||
self.backend_script(root)
|
||||
script = root / 'deploy/offline/control.py'
|
||||
script.write_text(textwrap.dedent('''
|
||||
import json, os, signal, subprocess, sys, time
|
||||
from pathlib import Path
|
||||
import psutil
|
||||
root = Path(__file__).resolve().parents[2]
|
||||
token = 'a' * 48
|
||||
java = root / 'runtimes/java/bin' / ('java.exe' if os.name == 'nt' else 'java')
|
||||
child = subprocess.Popen([str(java), str(root / 'backend-test.py'), '-jar', str(root / 'app/backend.jar')])
|
||||
def record(pid):
|
||||
process = psutil.Process(pid)
|
||||
return {'pid': pid, 'created': process.create_time(), 'executable': str(Path(process.exe()).resolve()), 'args': process.cmdline()[1:]}
|
||||
state = {'supervisor': record(os.getpid()), 'token': token, 'children': [{**record(child.pid), 'name': 'backend'}], 'ready': True}
|
||||
def stop(*ignored):
|
||||
child.terminate()
|
||||
child.wait(timeout=5)
|
||||
(root / '.runtime/supervisor-stopped').write_text('done')
|
||||
(root / '.runtime/service.json').unlink(missing_ok=True)
|
||||
raise SystemExit(0)
|
||||
signal.signal(signal.SIGTERM, stop)
|
||||
(root / '.runtime/service.json').write_text(json.dumps(state))
|
||||
(root / '.runtime/supervisor-ready').write_text('ready')
|
||||
while True:
|
||||
try:
|
||||
request = json.loads((root / '.runtime/stop-request.json').read_text())
|
||||
if request.get('token') == token:
|
||||
(root / '.runtime/token-observed').write_text(token)
|
||||
stop()
|
||||
except (FileNotFoundError, ValueError):
|
||||
pass
|
||||
time.sleep(.02)
|
||||
'''))
|
||||
process = subprocess.Popen([str(bootstrap.runtime_executable(root, 'supervisor')), str(script), 'serve'],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
self.addCleanup(self.stop_test_child, process)
|
||||
self.wait_file(root / '.runtime/supervisor-ready', process)
|
||||
self.wait_file(root / '.runtime/backend-ready', process)
|
||||
return process
|
||||
|
||||
def test_recovers_before_importing_broken_control_and_updater(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
root, directory, backup, journal, old = self.fixture(folder)
|
||||
result = bootstrap.recover(root)
|
||||
self.assertTrue(result['recovered'])
|
||||
self.assertEqual((root / '.runtime/restored').read_text(), 'done')
|
||||
for name, content in old.items():
|
||||
self.assertEqual((root / name).read_bytes(), content)
|
||||
self.assertFalse((directory / 'worker.json').exists())
|
||||
|
||||
def test_corrupt_backup_is_rejected_before_any_replacement(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
root, directory, backup, journal, old = self.fixture(folder)
|
||||
(backup / 'program/deploy/offline/control.py').write_text('corruption')
|
||||
with self.assertRaises(RuntimeError):
|
||||
bootstrap.recover(root)
|
||||
self.assertEqual((root / 'deploy/offline/control.py').read_text(), 'def broken syntax\n')
|
||||
|
||||
def test_active_worker_cannot_be_overridden(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
root, directory, backup, journal, old = self.fixture(folder)
|
||||
(directory / 'worker.json').write_text(json.dumps({'pid': os.getpid(), 'created': psutil.Process().create_time()}))
|
||||
with self.assertRaises(RuntimeError):
|
||||
bootstrap.recover(root)
|
||||
self.assertFalse((root / '.runtime/restored').exists())
|
||||
|
||||
def test_program_protection_and_no_update_are_read_only(self):
|
||||
for name in ('deploy/bootstrap/windows-start.ps1', 'deploy/offline/update-public-key.pem', '.runtime/deployment.json', 'runtimedata/mysql/file'):
|
||||
self.assertFalse(bootstrap.program(name))
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
root = Path(folder)
|
||||
self.assertEqual(bootstrap.recover(root), {'recovered': False})
|
||||
self.assertFalse((root / '.runtime').exists())
|
||||
|
||||
def test_orphaned_backend_stops_before_jar_copy_after_worker_crash(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
root, directory, _, _, old = self.fixture(folder)
|
||||
child = self.start_backend(root)
|
||||
state = {'supervisor': {'pid': 999999999, 'created': 1, 'executable': str(sys.executable),
|
||||
'args': [str(root / 'deploy/offline/control.py'), 'serve']}, 'token': 'b' * 48,
|
||||
'children': [self.record(child, 'backend')]}
|
||||
(root / '.runtime/service.json').write_text(json.dumps(state))
|
||||
(directory / 'worker.json').write_text(json.dumps({'pid': 999999998, 'created': 1}))
|
||||
before = bootstrap.digest(root / 'app/backend.jar')
|
||||
real_copy = bootstrap.atomic_copy
|
||||
def checked_copy(source, target):
|
||||
self.assertIsNotNone(child.poll(), 'A mutable program was copied while the old backend still held its JAR')
|
||||
return real_copy(source, target)
|
||||
with patch.object(bootstrap, 'atomic_copy', side_effect=checked_copy):
|
||||
self.assertTrue(bootstrap.recover(root)['recovered'])
|
||||
self.assertEqual(before, (root / '.runtime/backend-stopped-before-copy').read_text())
|
||||
self.assertEqual(old['app/backend.jar'], (root / 'app/backend.jar').read_bytes())
|
||||
|
||||
def test_live_supervisor_receives_matching_stop_token_and_children_exit_before_copy(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
root, directory, _, _, _ = self.fixture(folder)
|
||||
supervisor = self.start_supervisor(root)
|
||||
(root / 'deploy/offline/control.py').write_text('def broken syntax\n')
|
||||
before = bootstrap.digest(root / 'app/backend.jar')
|
||||
result = bootstrap.recover(root)
|
||||
self.assertTrue(result['recovered'])
|
||||
self.assertIsNotNone(supervisor.poll())
|
||||
self.assertEqual('a' * 48, (root / '.runtime/token-observed').read_text())
|
||||
self.assertEqual(before, (root / '.runtime/backend-stopped-before-copy').read_text())
|
||||
self.assertFalse((root / '.runtime/stop-request.json').exists())
|
||||
|
||||
def test_stale_record_pointing_at_an_unrelated_live_pid_never_kills_or_replaces(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
root, _, _, _, _ = self.fixture(folder)
|
||||
self.runtime_aliases(root)
|
||||
child = subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(60)'], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
self.addCleanup(self.stop_test_child, child)
|
||||
info = self.record(child, 'backend')
|
||||
info['created'] -= 60
|
||||
(root / '.runtime/service.json').write_text(json.dumps({'children': [info]}))
|
||||
with self.assertRaisesRegex(RuntimeError, '归属'):
|
||||
bootstrap.recover(root)
|
||||
self.assertIsNone(child.poll())
|
||||
self.assertEqual('def broken syntax\n', (root / 'deploy/offline/control.py').read_text())
|
||||
self.assertFalse((root / '.runtime/restored').exists())
|
||||
|
||||
def test_unregistered_root_backend_is_refused_without_signalling(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
root, _, _, _, _ = self.fixture(folder)
|
||||
child = self.start_backend(root)
|
||||
with self.assertRaisesRegex(RuntimeError, '未登记'):
|
||||
bootstrap.recover(root)
|
||||
self.assertIsNone(child.poll())
|
||||
self.assertEqual('def broken syntax\n', (root / 'deploy/offline/control.py').read_text())
|
||||
|
||||
def test_other_deployment_root_cannot_be_claimed_by_a_valid_live_pid_record(self):
|
||||
with tempfile.TemporaryDirectory() as first, tempfile.TemporaryDirectory() as second:
|
||||
root, _, _, _, _ = self.fixture(first)
|
||||
other, _, _, _, _ = self.fixture(second)
|
||||
self.runtime_aliases(root)
|
||||
child = self.start_backend(other)
|
||||
(root / '.runtime/service.json').write_text(json.dumps({'children': [self.record(child, 'backend')]}))
|
||||
with self.assertRaisesRegex(RuntimeError, '归属'):
|
||||
bootstrap.recover(root)
|
||||
self.assertIsNone(child.poll())
|
||||
self.assertFalse((root / '.runtime/restored').exists())
|
||||
|
||||
def test_invalid_supervisor_token_does_not_signal_any_owned_process(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
root, _, _, _, _ = self.fixture(folder)
|
||||
supervisor = self.start_supervisor(root)
|
||||
state = json.loads((root / '.runtime/service.json').read_text())
|
||||
state['token'] = 'invalid-token'
|
||||
(root / '.runtime/service.json').write_text(json.dumps(state))
|
||||
with self.assertRaisesRegex(RuntimeError, '令牌'):
|
||||
bootstrap.recover(root)
|
||||
self.assertIsNone(supervisor.poll())
|
||||
self.assertFalse((root / '.runtime/stop-request.json').exists())
|
||||
|
||||
def test_dead_pid_records_do_not_block_safe_recovery(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
root, _, _, _, _ = self.fixture(folder)
|
||||
state = {'children': [{'pid': 999999999, 'created': 1, 'executable': str(sys.executable),
|
||||
'args': ['missing'], 'name': 'backend'}]}
|
||||
(root / '.runtime/service.json').write_text(json.dumps(state))
|
||||
self.assertTrue(bootstrap.recover(root)['recovered'])
|
||||
|
||||
def test_a_process_owned_by_another_user_is_never_stopped(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
root, _, _, _, _ = self.fixture(folder)
|
||||
child = self.start_backend(root)
|
||||
(root / '.runtime/service.json').write_text(json.dumps({'children': [self.record(child, 'backend')]}))
|
||||
with patch.object(bootstrap, 'current_user', return_value=False), self.assertRaisesRegex(RuntimeError, '归属'):
|
||||
bootstrap.recover(root)
|
||||
self.assertIsNone(child.poll())
|
||||
self.assertFalse((root / '.runtime/restored').exists())
|
||||
|
||||
@unittest.skipIf(os.name == 'nt', 'Host-side immutable-client fixture uses a POSIX shebang; real Windows binaries need native acceptance')
|
||||
def test_orphaned_mysql_uses_immutable_shutdown_client_and_is_not_force_killed(self):
|
||||
with tempfile.TemporaryDirectory() as folder:
|
||||
root, _, _, _, _ = self.fixture(folder)
|
||||
executable = bootstrap.runtime_executable(root, 'mysql')
|
||||
executable.parent.mkdir(parents=True)
|
||||
executable.symlink_to(psutil.Process().exe())
|
||||
server = root / 'mysql-test.py'
|
||||
server.write_text(textwrap.dedent('''
|
||||
import os, signal, time
|
||||
from pathlib import Path
|
||||
root = Path(__file__).parent
|
||||
def stop(*ignored):
|
||||
(root / '.runtime/mysql-stopped-cleanly').write_text('done')
|
||||
raise SystemExit(0)
|
||||
signal.signal(signal.SIGTERM, stop)
|
||||
(root / '.runtime/mysql-test-pid').write_text(str(os.getpid()))
|
||||
while True:
|
||||
time.sleep(.02)
|
||||
'''))
|
||||
child = subprocess.Popen([str(executable), str(server), '--datadir=' + str(root / 'runtimedata/mysql'), '--port=54489'],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
self.addCleanup(self.stop_test_child, child)
|
||||
self.wait_file(root / '.runtime/mysql-test-pid', child)
|
||||
admin = executable.parent / 'mysqladmin'
|
||||
admin.write_text('#!' + psutil.Process().exe() + '\n' + textwrap.dedent('''
|
||||
import os, signal, sys
|
||||
from pathlib import Path
|
||||
root = Path(__file__).resolve().parents[3]
|
||||
assert os.environ['MYSQL_PWD'] == 'isolated_password_value_1234'
|
||||
assert '--no-defaults' in sys.argv and '--port=54489' in sys.argv and sys.argv[-1] == 'shutdown'
|
||||
assert not any('password' in arg.lower() for arg in sys.argv)
|
||||
(root / '.runtime/immutable-mysqladmin-called').write_text('done')
|
||||
os.kill(int((root / '.runtime/mysql-test-pid').read_text()), signal.SIGTERM)
|
||||
'''))
|
||||
admin.chmod(0o700)
|
||||
(root / '.runtime/deployment.json').write_text(json.dumps({'mysqlPort': 54489, 'rootPassword': 'isolated_password_value_1234'}))
|
||||
(root / '.runtime/service.json').write_text(json.dumps({'children': [self.record(child, 'mysql')]}))
|
||||
with patch.object(psutil.Process, 'kill', side_effect=AssertionError('MySQL must not be force killed')):
|
||||
self.assertTrue(bootstrap.recover(root)['recovered'])
|
||||
self.assertIsNotNone(child.poll())
|
||||
self.assertTrue((root / '.runtime/immutable-mysqladmin-called').exists())
|
||||
self.assertTrue((root / '.runtime/mysql-stopped-cleanly').exists())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,303 @@
|
||||
"""Offline security and rollback tests. No network, production DB or services."""
|
||||
import base64
|
||||
import copy
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import stat
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
import zipfile
|
||||
|
||||
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
|
||||
from cryptography.hazmat.primitives import serialization
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(HERE))
|
||||
import update_manager as update
|
||||
|
||||
|
||||
class UpdateTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name) / 'bundle'
|
||||
self.root.mkdir()
|
||||
self.private = Ed25519PrivateKey.generate()
|
||||
self.files = {
|
||||
'app/backend.jar': b'old backend',
|
||||
'app/web/index.html': b'old web',
|
||||
'app/web/obsolete.js': b'old asset',
|
||||
'deploy/offline/control.py': b'old supervisor',
|
||||
'deploy/offline/update_manager.py': b'old updater',
|
||||
'backend/src/main/resources/analysis-runtime-policy.json': b'{}',
|
||||
update.KEY: self.private.public_key().public_bytes(serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo),
|
||||
'runtimes/python/python.exe': b'unchanged runtime',
|
||||
'migration/snapshot.sql': b'private migration data',
|
||||
}
|
||||
for name, data in self.files.items():
|
||||
self.write(name, data)
|
||||
self.manifest = {
|
||||
'platform': 'windows', 'architecture': 'x86_64',
|
||||
'runtimes': [{'name': 'python', 'version': '3.11.15'}],
|
||||
'pythonWheels': [{'file': 'cryptography.whl', 'sha256': 'a' * 64}],
|
||||
'files': [{'path': k, 'sha256': hashlib.sha256(v).hexdigest()} for k, v in self.files.items()],
|
||||
'updateSequence': 5, 'updateVersion': '2026.11.03',
|
||||
}
|
||||
update.atomic(self.root / 'manifest.json', self.manifest)
|
||||
self.config = {'mysqlDatabase': 'quality_restore_' + 'a' * 24, 'archiveDirectory': 'runtimedata/archive/restore-old',
|
||||
'mysqlPort': 53306, 'rootPassword': 'privateRootPasswordValue', 'mysqlPassword': 'privateAppPasswordValue'}
|
||||
update.atomic(self.root / '.runtime/deployment.json', self.config)
|
||||
self.write('.runtime/analysis-api-settings.json', b'{"apiKey":"keep-private"}')
|
||||
self.program = {k: b'new ' + v for k, v in self.files.items() if update.program_path(k) and k != 'app/web/obsolete.js'}
|
||||
self.release = {'schema': 1, 'version': '2026.11.04', 'sequence': 6, 'platform': 'windows-x64',
|
||||
'runtimeFingerprint': update.runtime_fingerprint(self.manifest),
|
||||
'archive': {'file': 'quality-update-2026.11.04.zip', 'size': 123, 'sha256': 'a' * 64},
|
||||
'files': [{'path': k, 'size': len(v), 'sha256': hashlib.sha256(v).hexdigest()} for k, v in self.program.items()]}
|
||||
|
||||
def write(self, name, data):
|
||||
path = self.root / name
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(data)
|
||||
|
||||
def envelope(self, release=None, private=None):
|
||||
raw = update.canonical(release or self.release)
|
||||
return {'payload': base64.b64encode(raw).decode(), 'signature': base64.b64encode((private or self.private).sign(raw)).decode()}
|
||||
|
||||
def validate(self, release=None):
|
||||
return update.validate_release(self.envelope(release), self.root)
|
||||
|
||||
def archive(self, entries=None):
|
||||
archive = Path(self.temp.name) / 'update.zip'
|
||||
with zipfile.ZipFile(archive, 'w') as out:
|
||||
for name, data in (entries or self.program).items():
|
||||
out.writestr(name, data)
|
||||
return archive
|
||||
|
||||
def test_valid_signature_and_runtime_identity(self):
|
||||
self.assertEqual('2026.11.04', self.validate()['version'])
|
||||
self.assertEqual(update.runtime_fingerprint(self.manifest), update.runtime_fingerprint(copy.deepcopy(self.manifest)))
|
||||
|
||||
def test_wrong_signature_key_and_modified_payload_are_rejected(self):
|
||||
with self.assertRaises(update.UpdateError):
|
||||
update.validate_release(self.envelope(private=Ed25519PrivateKey.generate()), self.root)
|
||||
envelope = self.envelope()
|
||||
envelope['payload'] = base64.b64encode(update.canonical({**self.release, 'sequence': 999})).decode()
|
||||
with self.assertRaises(update.UpdateError):
|
||||
update.validate_release(envelope, self.root)
|
||||
|
||||
def test_immutable_trust_anchor_checked_against_local_manifest(self):
|
||||
self.write(update.KEY, Ed25519PrivateKey.generate().public_key().public_bytes(serialization.Encoding.PEM, serialization.PublicFormat.SubjectPublicKeyInfo))
|
||||
with self.assertRaises(update.UpdateError):
|
||||
self.validate()
|
||||
|
||||
def test_runtime_mismatch_requires_full_bundle(self):
|
||||
release = {**self.release, 'runtimeFingerprint': 'b' * 64}
|
||||
with self.assertRaisesRegex(update.UpdateError, '完整安装包'):
|
||||
self.validate(release)
|
||||
|
||||
def test_forbidden_destinations_and_trust_key_are_never_update_files(self):
|
||||
for name in ['.runtime/deployment.json', 'runtimedata/mysql/table.ibd', 'migration/snapshot.sql',
|
||||
'runtimes/python/python.exe', update.KEY, 'manifest.json', 'backend/src/private.key']:
|
||||
with self.subTest(name=name):
|
||||
release = copy.deepcopy(self.release)
|
||||
release['files'].append({'path': name, 'size': 0, 'sha256': 'a' * 64})
|
||||
with self.assertRaises(update.UpdateError):
|
||||
self.validate(release)
|
||||
|
||||
def test_windows_aliases_traversal_and_case_collisions_rejected(self):
|
||||
for name in ['../evil', 'app/web/../evil', 'app/web/C:evil', 'app\\web\\evil', '/absolute',
|
||||
'app/web/NUL.txt', 'app/web/trailing.', 'app/web/double//slash', 'app/web/control\x01']:
|
||||
with self.subTest(name=name), self.assertRaises(update.UpdateError):
|
||||
update.safe_path(name)
|
||||
release = copy.deepcopy(self.release)
|
||||
release['files'].append({**release['files'][0], 'path': release['files'][0]['path'].upper()})
|
||||
with self.assertRaises(update.UpdateError):
|
||||
self.validate(release)
|
||||
|
||||
def test_destination_symlink_rejected(self):
|
||||
target = self.root / 'app/web/link'
|
||||
target.symlink_to(Path(self.temp.name), target_is_directory=True)
|
||||
with self.assertRaises(update.UpdateError):
|
||||
update.destination(self.root, 'app/web/link/escape.html')
|
||||
|
||||
def test_missing_required_program_or_deployment_manifest_rejected(self):
|
||||
release = copy.deepcopy(self.release)
|
||||
release['files'] = [e for e in release['files'] if e['path'] != 'backend/src/main/resources/analysis-runtime-policy.json']
|
||||
with self.assertRaises(update.UpdateError):
|
||||
self.validate(release)
|
||||
with self.assertRaisesRegex(update.UpdateError, '部署数据'):
|
||||
self.validate({**self.release, 'bundleManifest': self.manifest})
|
||||
|
||||
def test_zip_inventory_and_hash_enforced(self):
|
||||
stage = Path(self.temp.name) / 'good'
|
||||
update.unpack(self.archive(), stage, self.release)
|
||||
self.assertEqual(self.program['app/backend.jar'], (stage / 'app/backend.jar').read_bytes())
|
||||
bad = dict(self.program)
|
||||
bad['../evil'] = b'escape'
|
||||
with self.assertRaises(update.UpdateError):
|
||||
update.unpack(self.archive(bad), Path(self.temp.name) / 'escape', self.release)
|
||||
bad = dict(self.program)
|
||||
bad['app/backend.jar'] = b'x' * len(bad['app/backend.jar'])
|
||||
with self.assertRaisesRegex(update.UpdateError, '校验失败'):
|
||||
update.unpack(self.archive(bad), Path(self.temp.name) / 'hash', self.release)
|
||||
|
||||
def test_zip_symlink_rejected(self):
|
||||
archive = Path(self.temp.name) / 'symlink.zip'
|
||||
info = zipfile.ZipInfo('app/backend.jar')
|
||||
info.external_attr = (stat.S_IFLNK | 0o777) << 16
|
||||
with zipfile.ZipFile(archive, 'w') as out:
|
||||
out.writestr(info, self.program['app/backend.jar'])
|
||||
with self.assertRaises(update.UpdateError):
|
||||
update.unpack(archive, Path(self.temp.name) / 'links', self.release)
|
||||
|
||||
def test_snapshot_replace_preserves_local_data_and_manifest_entries(self):
|
||||
backup = self.root / '.runtime/update/backup-test'
|
||||
backup.mkdir(parents=True)
|
||||
stage = Path(self.temp.name) / 'stage'
|
||||
update.unpack(self.archive(), stage, self.release)
|
||||
update.backup_program(self.root, backup, self.release)
|
||||
update.replace_program(self.root, stage, backup, self.release)
|
||||
result = update.read(self.root / 'manifest.json')
|
||||
self.assertEqual(6, result['updateSequence'])
|
||||
self.assertEqual(self.config, update.read(self.root / '.runtime/deployment.json'))
|
||||
self.assertFalse((self.root / 'app/web/obsolete.js').exists())
|
||||
for name in (update.KEY, 'runtimes/python/python.exe', 'migration/snapshot.sql'):
|
||||
self.assertEqual(self.files[name], (self.root / name).read_bytes())
|
||||
self.assertIn(next(e for e in self.manifest['files'] if e['path'] == name), result['files'])
|
||||
update.restore_program(self.root, backup)
|
||||
self.assertEqual(self.manifest, update.read(self.root / 'manifest.json'))
|
||||
self.assertEqual(self.files['app/web/obsolete.js'], (self.root / 'app/web/obsolete.js').read_bytes())
|
||||
|
||||
def worker_harness(self, fail_verify=False):
|
||||
directory = self.root / '.runtime/update'
|
||||
update.atomic(directory / 'request.json', {'token': 'a' * 32})
|
||||
archive = self.archive()
|
||||
envelope = self.envelope()
|
||||
original_download = update.download
|
||||
calls, restores, queries = [], [], []
|
||||
|
||||
def fake_download(url, target=None, *args):
|
||||
if target:
|
||||
shutil.copyfile(archive, target)
|
||||
return b''
|
||||
return json.dumps(envelope).encode()
|
||||
|
||||
def fake_control(root, action, database_only=False, timeout=0):
|
||||
calls.append((action, database_only))
|
||||
if action == 'verify' and fail_verify:
|
||||
raise update.UpdateError('模拟新版启动校验失败。')
|
||||
return types.SimpleNamespace(pid=1234)
|
||||
|
||||
def fake_mysql(root, config, binary='mysql', database=None, stdin=None, stdout=None):
|
||||
if binary == 'mysqldump':
|
||||
stdout.write(b'-- database snapshot\n' + b'SELECT 1;\n' * 30)
|
||||
else:
|
||||
restores.append((database, stdin.read()))
|
||||
|
||||
process = types.SimpleNamespace(parents=lambda: [], create_time=lambda: 1,
|
||||
is_running=lambda: True, status=lambda: 'running')
|
||||
with patch.object(update.psutil, 'pid_exists', return_value=False), \
|
||||
patch.object(update.psutil, 'Process', return_value=process), \
|
||||
patch.object(update, 'download', side_effect=fake_download), \
|
||||
patch.object(update, 'controller', side_effect=fake_control), \
|
||||
patch.object(update, 'wait_ready'), \
|
||||
patch.object(update, 'mysql', side_effect=fake_mysql), \
|
||||
patch.object(update, 'sql', side_effect=lambda root, config, query: queries.append(query)):
|
||||
update.worker(987654, 'a' * 32, self.root)
|
||||
return calls, restores, queries
|
||||
|
||||
def test_complete_update_orchestration_with_isolated_database_backup(self):
|
||||
calls, restores, queries = self.worker_harness()
|
||||
self.assertEqual([('stop', False), ('serve', True), ('stop', False), ('verify', False), ('serve', False)], calls)
|
||||
self.assertFalse(restores)
|
||||
self.assertEqual('completed', update.read(self.root / '.runtime/update/status.json')['state'])
|
||||
self.assertEqual('complete', update.read(self.root / '.runtime/update/journal.json')['phase'])
|
||||
self.assertEqual(6, update.read(self.root / 'manifest.json')['updateSequence'])
|
||||
|
||||
def test_failed_update_restores_old_program_and_new_rollback_database(self):
|
||||
self.config['restoredFromMigration'] = 'isolated-test-migration'
|
||||
update.atomic(self.root / '.runtime/deployment.json', self.config)
|
||||
update.atomic(self.root / '.runtime/migration-receipt.json', {'migrationId': 'isolated-test-migration',
|
||||
'database': self.config['mysqlDatabase'], 'migrationManifestSha256': 'b' * 64})
|
||||
calls, restores, queries = self.worker_harness(fail_verify=True)
|
||||
self.assertEqual(self.manifest, update.read(self.root / 'manifest.json'))
|
||||
config = update.read(self.root / '.runtime/deployment.json')
|
||||
self.assertRegex(config['mysqlDatabase'], r'^quality_restore_[a-f0-9]{24}$')
|
||||
self.assertNotEqual(self.config['mysqlDatabase'], config['mysqlDatabase'])
|
||||
self.assertEqual(self.config['archiveDirectory'], config['archiveDirectory'])
|
||||
self.assertEqual(config['mysqlDatabase'], restores[0][0])
|
||||
self.assertEqual(config['mysqlDatabase'], update.read(self.root / '.runtime/migration-receipt.json')['database'])
|
||||
self.assertTrue(any('CREATE DATABASE' in q for q in queries))
|
||||
self.assertFalse(any('DROP' in q for q in queries))
|
||||
self.assertEqual('failed', update.read(self.root / '.runtime/update/status.json')['state'])
|
||||
self.assertIn('已恢复原版本', update.read(self.root / '.runtime/update/status.json')['message'])
|
||||
self.assertEqual('rolledBack', update.read(self.root / '.runtime/update/journal.json')['phase'])
|
||||
|
||||
def test_recovery_blocks_live_worker_and_rejects_untrusted_backup_path(self):
|
||||
directory = self.root / '.runtime/update'
|
||||
update.atomic(directory / 'journal.json', {'phase': 'installing', 'backup': '../../escape'})
|
||||
with patch.object(update, 'live', return_value=False), self.assertRaises(update.UpdateError):
|
||||
update.recover(self.root)
|
||||
with patch.object(update, 'live', return_value=True), self.assertRaisesRegex(update.UpdateError, '正在进行'):
|
||||
update.recover(self.root)
|
||||
|
||||
def test_arbitrary_download_url_is_rejected_before_network(self):
|
||||
for url in ['http://quality.lyunlab.me/quality-updates/windows-x64/latest.json', 'https://evil.example/update.zip',
|
||||
update.BASE + '../private.zip', update.BASE + 'quality-update-x.zip?token=x']:
|
||||
with self.subTest(url=url), self.assertRaises(update.UpdateError):
|
||||
update.download(url)
|
||||
|
||||
def test_source_is_disabled_and_status_never_exposes_private_fields(self):
|
||||
update.atomic(self.root / '.runtime/update/status.json', {'state': 'idle', 'rootPassword': 'secret', 'backupPath': '/private'})
|
||||
with patch.object(update.os, 'name', 'posix'):
|
||||
self.assertFalse(update.enabled(self.root))
|
||||
result = update.status(self.root)
|
||||
self.assertNotIn('rootPassword', result)
|
||||
self.assertNotIn('backupPath', result)
|
||||
|
||||
def test_recovery_and_worker_share_lock_and_live_identity(self):
|
||||
with update.exclusive_update(self.root):
|
||||
self.assertTrue(update.live(update.read(self.root / '.runtime/update/worker.json')))
|
||||
with self.assertRaises(update.BusyUpdate):
|
||||
update.recover(self.root)
|
||||
# The lock remains authoritative if the identity file disappears.
|
||||
(self.root / '.runtime/update/worker.json').unlink()
|
||||
with self.assertRaises(update.BusyUpdate):
|
||||
with update.exclusive_update(self.root):
|
||||
self.fail('Concurrent update lock was acquired')
|
||||
|
||||
def test_atomic_copy_failure_preserves_existing_control_and_configuration(self):
|
||||
source = Path(self.temp.name) / 'new-content'
|
||||
source.write_bytes(b'new file content')
|
||||
for name in ('deploy/offline/control.py', '.runtime/deployment.json', 'manifest.json'):
|
||||
target = self.root / name
|
||||
previous = target.read_bytes()
|
||||
with self.subTest(name=name), patch.object(update.os, 'replace', side_effect=OSError('simulated interruption')):
|
||||
with self.assertRaises(OSError):
|
||||
update.copy_atomic(source, target)
|
||||
self.assertEqual(previous, target.read_bytes())
|
||||
self.assertFalse(list(target.parent.glob(target.name + '.update-*')))
|
||||
|
||||
def test_mysqldump_does_not_receive_mysql_only_connect_timeout_option(self):
|
||||
with patch.object(update.subprocess, 'run', return_value=types.SimpleNamespace(returncode=0)) as run:
|
||||
update.mysql(self.root, self.config, 'mysqldump', stdout=update.subprocess.DEVNULL)
|
||||
self.assertFalse(any(arg.startswith('--connect-timeout') for arg in run.call_args.args[0]))
|
||||
update.mysql(self.root, self.config, 'mysql')
|
||||
self.assertIn('--connect-timeout=10', run.call_args.args[0])
|
||||
|
||||
def test_actual_dump_is_followed_by_a_second_recovery_space_check(self):
|
||||
with patch.object(update.shutil, 'disk_usage', side_effect=[types.SimpleNamespace(free=10**15), types.SimpleNamespace(free=0)]):
|
||||
calls, _, _ = self.worker_harness()
|
||||
self.assertNotIn(('verify', False), calls)
|
||||
self.assertEqual(self.manifest, update.read(self.root / 'manifest.json'))
|
||||
self.assertIn('剩余磁盘空间不足', update.read(self.root / '.runtime/update/status.json')['message'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,3 @@
|
||||
-----BEGIN PUBLIC KEY-----
|
||||
MCowBQYDK2VwAyEAMPNTCeYpWygXdSxtaJzijXu0mKZF64lP7NcUuYZ4NcU=
|
||||
-----END PUBLIC KEY-----
|
||||
@@ -0,0 +1,357 @@
|
||||
"""Immutable stage-zero recovery. Installed under deploy/bootstrap, never updated."""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import secrets
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
import psutil # Bundled immutable Python environment, not an application module.
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def read(path, default=None):
|
||||
return json.loads(path.read_text(encoding='utf-8')) if path.exists() else default
|
||||
|
||||
|
||||
def digest(path):
|
||||
with path.open('rb') as stream:
|
||||
return hashlib.file_digest(stream, 'sha256').hexdigest()
|
||||
|
||||
|
||||
def atomic_copy(source, target):
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = target.with_name(target.name + '.recover-' + secrets.token_hex(6))
|
||||
try:
|
||||
with source.open('rb') as reader, temporary.open('xb') as writer:
|
||||
shutil.copyfileobj(reader, writer)
|
||||
writer.flush()
|
||||
os.fsync(writer.fileno())
|
||||
temporary.replace(target)
|
||||
finally:
|
||||
temporary.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def safe(root, name):
|
||||
if (not isinstance(name, str) or not name or '\\' in name or ':' in name or name.startswith('/')
|
||||
or any(part in ('', '.', '..') or part.endswith((' ', '.')) for part in name.split('/'))):
|
||||
raise RuntimeError('更新恢复路径无效。')
|
||||
target = root.joinpath(*name.split('/'))
|
||||
for path in (target, *target.parents):
|
||||
if path == root:
|
||||
break
|
||||
if path.is_symlink() or (path.exists() and getattr(path.lstat(), 'st_file_attributes', 0) & 0x400):
|
||||
raise RuntimeError('更新恢复目录包含无效链接。')
|
||||
if not target.resolve().is_relative_to(root.resolve()):
|
||||
raise RuntimeError('更新恢复路径越界。')
|
||||
return target
|
||||
|
||||
|
||||
def program(name):
|
||||
if name == 'deploy/offline/update-public-key.pem' or name.startswith('deploy/bootstrap/'):
|
||||
return False
|
||||
path = Path(name)
|
||||
return (name == 'app/backend.jar' or name.startswith('app/web/')
|
||||
or name == 'backend/src/main/resources/analysis-runtime-policy.json'
|
||||
or name.startswith('scripts/') and path.suffix == '.py'
|
||||
or name.startswith('ai/nao-quality/')
|
||||
or name.startswith('deploy/offline/') and path.suffix in ('.py', '.ps1', '.md'))
|
||||
|
||||
|
||||
def atomic_json(path, value):
|
||||
temporary = path.with_name(path.name + '.stop-' + secrets.token_hex(6))
|
||||
try:
|
||||
with temporary.open('x', encoding='utf-8') as stream:
|
||||
json.dump(value, stream)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
temporary.replace(path)
|
||||
finally:
|
||||
temporary.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def same_path(left, right):
|
||||
return os.path.normcase(str(Path(left).resolve())) == os.path.normcase(str(Path(right).resolve()))
|
||||
|
||||
|
||||
def runtime_executable(root, role):
|
||||
suffix = '.exe' if os.name == 'nt' else ''
|
||||
relative = {'supervisor': 'runtimes/python/' + ('python.exe' if os.name == 'nt' else 'bin/python3'),
|
||||
'backend': 'runtimes/java/bin/java' + suffix, 'mysql': 'runtimes/mysql/bin/mysqld' + suffix}[role]
|
||||
return root / relative
|
||||
|
||||
|
||||
def bound_to_root(root, role, args):
|
||||
if not isinstance(args, list) or not all(isinstance(arg, str) for arg in args):
|
||||
return False
|
||||
if role == 'supervisor':
|
||||
return (len(args) in (2, 3) and same_path(args[0], root / 'deploy/offline/control.py')
|
||||
and args[1] == 'serve' and (len(args) == 2 or args[2] == '--database-only'))
|
||||
if role == 'backend':
|
||||
return len(args) >= 2 and args[-2] == '-jar' and same_path(args[-1], root / 'app/backend.jar')
|
||||
datadirs = [arg.split('=', 1)[1] for arg in args if arg.startswith('--datadir=')]
|
||||
return len(datadirs) == 1 and same_path(datadirs[0], root / 'runtimedata/mysql')
|
||||
|
||||
|
||||
def current_user(process):
|
||||
if os.name == 'nt':
|
||||
return process.username().casefold() == psutil.Process().username().casefold()
|
||||
return process.uids().real == os.getuid()
|
||||
|
||||
|
||||
def owned_process(root, role, info):
|
||||
"""Return a fully verified owned PID, None for a dead PID, never guess."""
|
||||
if (not isinstance(info, dict) or type(info.get('pid')) is not int or info['pid'] <= 0
|
||||
or not isinstance(info.get('created'), (int, float))
|
||||
or not isinstance(info.get('executable'), str) or not isinstance(info.get('args'), list)):
|
||||
raise RuntimeError('本包服务进程记录无效,未停止任何不明进程。')
|
||||
try:
|
||||
process = psutil.Process(info['pid'])
|
||||
if not process.is_running() or process.status() == psutil.STATUS_ZOMBIE:
|
||||
return None
|
||||
forbidden = {os.getpid(), *(parent.pid for parent in psutil.Process().parents())}
|
||||
if (process.pid in forbidden or abs(process.create_time() - info['created']) >= .01
|
||||
or not current_user(process) or not same_path(process.exe(), info['executable'])
|
||||
or process.cmdline()[1:] != info['args']
|
||||
or not same_path(info['executable'], runtime_executable(root, role))
|
||||
or not bound_to_root(root, role, info['args'])):
|
||||
raise RuntimeError('无法确认本包服务进程归属,未停止任何不明进程。')
|
||||
return process
|
||||
except psutil.NoSuchProcess:
|
||||
return None
|
||||
except psutil.Error:
|
||||
raise RuntimeError('当前用户无法核验本包服务进程,恢复已安全停止。') from None
|
||||
|
||||
|
||||
def root_markers(root, args):
|
||||
"""Detect unregistered package processes without inspecting unrelated data."""
|
||||
for arg in args:
|
||||
if arg.startswith('--datadir='):
|
||||
if same_path(arg.split('=', 1)[1], root / 'runtimedata/mysql'):
|
||||
return True
|
||||
elif Path(arg).is_absolute() and (same_path(arg, root / 'app/backend.jar')
|
||||
or same_path(arg, root / 'deploy/offline/control.py')):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def refuse_unregistered(root, expected):
|
||||
for process in psutil.process_iter():
|
||||
try:
|
||||
if process.pid == os.getpid() or process.status() == psutil.STATUS_ZOMBIE:
|
||||
continue
|
||||
args = process.cmdline()[1:]
|
||||
if not root_markers(root, args):
|
||||
continue
|
||||
if process.pid not in expected:
|
||||
raise RuntimeError('发现未登记的本包服务进程,已保留现场,请先检查该进程。')
|
||||
except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess):
|
||||
continue
|
||||
|
||||
|
||||
def wait_owned(root, role, info, seconds):
|
||||
deadline = time.monotonic() + seconds
|
||||
while time.monotonic() < deadline:
|
||||
if owned_process(root, role, info) is None:
|
||||
return True
|
||||
time.sleep(.05)
|
||||
return owned_process(root, role, info) is None
|
||||
|
||||
|
||||
def terminate_owned(root, role, info, seconds):
|
||||
process = owned_process(root, role, info)
|
||||
if process is None:
|
||||
return
|
||||
try:
|
||||
process.terminate()
|
||||
if wait_owned(root, role, info, seconds):
|
||||
return
|
||||
process = owned_process(root, role, info)
|
||||
if process is not None:
|
||||
process.kill()
|
||||
if not wait_owned(root, role, info, 10):
|
||||
raise RuntimeError('本包服务尚未退出,未替换任何程序文件。')
|
||||
except psutil.NoSuchProcess:
|
||||
return
|
||||
except psutil.Error:
|
||||
raise RuntimeError('当前用户无法停止本包服务,未替换任何程序文件。') from None
|
||||
|
||||
|
||||
def shutdown_owned_mysql(root, info):
|
||||
process = owned_process(root, 'mysql', info)
|
||||
if process is None:
|
||||
return
|
||||
value = read(safe(root, '.runtime/deployment.json'), {})
|
||||
port = value.get('mysqlPort')
|
||||
if (type(port) is not int or not 1024 <= port <= 65535
|
||||
or not re.fullmatch(r'[A-Za-z0-9_-]{20,100}', str(value.get('rootPassword', '')))
|
||||
or info['args'].count('--port=' + str(port)) != 1):
|
||||
raise RuntimeError('无法核验本包数据库关闭配置,未替换任何程序文件。')
|
||||
executable = root / 'runtimes/mysql/bin' / ('mysqladmin.exe' if os.name == 'nt' else 'mysqladmin')
|
||||
env = os.environ.copy()
|
||||
env['MYSQL_PWD'] = value['rootPassword']
|
||||
# Only the immutable bundled MySQL client is executed; no application code
|
||||
# is loaded, and the current user's privileges are never elevated.
|
||||
command = [str(executable), '--no-defaults', '--protocol=TCP', '--host=127.0.0.1',
|
||||
'--port=' + str(port), '--user=root', '--connect-timeout=3', 'shutdown']
|
||||
try:
|
||||
subprocess.run(command, env=env, cwd=root, stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=20)
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
raise RuntimeError('本包数据库无法安全关闭,未替换任何程序文件。') from None
|
||||
if not wait_owned(root, 'mysql', info, 120):
|
||||
raise RuntimeError('本包数据库仍在关闭,未替换任何程序文件,请稍后重新启动。')
|
||||
|
||||
|
||||
def stop_owned_services(root, grace_seconds=200):
|
||||
"""Immutable implementation of service.json + stop-request token contract.
|
||||
|
||||
A crashed worker can leave a healthy new supervisor/Java behind. Validate
|
||||
every recorded identity before requesting shutdown; a live stale/reused PID
|
||||
is an error rather than permission to terminate it. Java must be gone before
|
||||
a JAR is replaced. An orphaned MySQL is shut down through its immutable
|
||||
client, never forcibly killed on Windows.
|
||||
"""
|
||||
state = read(safe(root, '.runtime/service.json'), {})
|
||||
if not isinstance(state, dict) or not isinstance(state.get('children', []), list):
|
||||
raise RuntimeError('本包服务状态无效,未替换任何程序文件。')
|
||||
entries = []
|
||||
supervisor = state.get('supervisor')
|
||||
if supervisor:
|
||||
entries.append(('supervisor', supervisor))
|
||||
for info in state.get('children', []):
|
||||
if not isinstance(info, dict) or info.get('name') not in ('backend', 'mysql'):
|
||||
raise RuntimeError('本包服务子进程记录无效,未替换任何程序文件。')
|
||||
entries.append((info['name'], info))
|
||||
if len(entries) > 20 or len({info.get('pid') for _, info in entries}) != len(entries):
|
||||
raise RuntimeError('本包服务进程记录重复,未替换任何程序文件。')
|
||||
# Validate all owners before the first stop request or signal is sent.
|
||||
active = {(role, info['pid']): owned_process(root, role, info) for role, info in entries}
|
||||
refuse_unregistered(root, {pid for (_, pid), process in active.items() if process is not None})
|
||||
if supervisor and active.get(('supervisor', supervisor['pid'])) is not None:
|
||||
token = state.get('token')
|
||||
if not isinstance(token, str) or not re.fullmatch(r'[a-f0-9]{48}', token):
|
||||
raise RuntimeError('本包服务停止令牌无效,未停止任何进程。')
|
||||
atomic_json(safe(root, '.runtime/stop-request.json'), {'token': token})
|
||||
if not wait_owned(root, 'supervisor', supervisor, grace_seconds):
|
||||
terminate_owned(root, 'supervisor', supervisor, 10)
|
||||
# A dead supervisor cannot finish its own finally block. Only independently
|
||||
# verified package children may be stopped; unrelated/reused PIDs are refused.
|
||||
for role, info in entries:
|
||||
if role == 'backend':
|
||||
terminate_owned(root, role, info, 35)
|
||||
for role, info in entries:
|
||||
if role == 'mysql':
|
||||
shutdown_owned_mysql(root, info)
|
||||
for role, info in entries:
|
||||
if owned_process(root, role, info) is not None:
|
||||
raise RuntimeError('本包服务尚未完全退出,未替换任何程序文件。')
|
||||
refuse_unregistered(root, set())
|
||||
stop = safe(root, '.runtime/stop-request.json')
|
||||
if isinstance(state.get('token'), str) and read(stop, {}).get('token') == state['token']:
|
||||
stop.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def restore_program_before_import(root, directory, journal):
|
||||
name = journal.get('backup', '')
|
||||
if not re.fullmatch(r'backup-[0-9]{8}T[0-9]{6}Z-[a-f0-9]{8}', name):
|
||||
raise RuntimeError('更新恢复记录无效。')
|
||||
backup = safe(directory, name)
|
||||
manifest_file, inventory_file = backup / 'manifest.json', backup / 'program.json'
|
||||
if (digest(manifest_file) != journal.get('backupManifestSha256')
|
||||
or digest(inventory_file) != journal.get('programInventorySha256')):
|
||||
raise RuntimeError('本机更新备份校验失败,请保留目录并联系管理员。')
|
||||
manifest = read(manifest_file)
|
||||
old = {entry['path']: entry for entry in manifest['files'] if program(entry['path'])}
|
||||
inventory = read(inventory_file)
|
||||
all_names, present = inventory['all'], set(inventory['present'])
|
||||
if (not isinstance(all_names, list) or len(all_names) > 50000
|
||||
or len(set(all_names)) != len(all_names) or not present.issubset(all_names)):
|
||||
raise RuntimeError('更新恢复清单无效。')
|
||||
for name in all_names:
|
||||
if not program(name):
|
||||
raise RuntimeError('更新恢复清单包含受保护路径。')
|
||||
target = safe(root, name)
|
||||
if name in present:
|
||||
source = safe(backup / 'program', name)
|
||||
if name not in old or not source.is_file() or digest(source) != old[name]['sha256']:
|
||||
raise RuntimeError('原程序备份缺失或损坏。')
|
||||
elif name in old:
|
||||
raise RuntimeError('原程序备份不完整。')
|
||||
# All backups are validated before any application file is replaced.
|
||||
stop_owned_services(root)
|
||||
for name in all_names:
|
||||
target = safe(root, name)
|
||||
if name in present:
|
||||
atomic_copy(safe(backup / 'program', name), target)
|
||||
else:
|
||||
target.unlink(missing_ok=True)
|
||||
atomic_copy(manifest_file, root / 'manifest.json')
|
||||
|
||||
|
||||
def live(info):
|
||||
try:
|
||||
process = psutil.Process(info['pid'])
|
||||
return (process.is_running() and process.status() != psutil.STATUS_ZOMBIE
|
||||
and abs(process.create_time() - info['created']) < .01)
|
||||
except (psutil.Error, ValueError, TypeError, KeyError):
|
||||
return False
|
||||
|
||||
|
||||
def recover(root=ROOT):
|
||||
directory = safe(root, '.runtime/update')
|
||||
journal = read(directory / 'journal.json', {})
|
||||
if journal.get('phase') in (None, 'complete', 'rolledBack'):
|
||||
return {'recovered': False}
|
||||
if live(read(directory / 'worker.json', {})):
|
||||
raise RuntimeError('更新正在进行,请等待原窗口完成。')
|
||||
with (directory / 'worker.lock').open('a+b') as lock:
|
||||
try:
|
||||
if os.name == 'nt':
|
||||
import msvcrt
|
||||
lock.seek(0)
|
||||
msvcrt.locking(lock.fileno(), msvcrt.LK_NBLCK, 1)
|
||||
else:
|
||||
import fcntl
|
||||
fcntl.flock(lock.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except OSError:
|
||||
raise RuntimeError('已有更新恢复正在进行,请等待完成。') from None
|
||||
identity = {'pid': os.getpid(), 'created': psutil.Process().create_time()}
|
||||
worker = directory / 'worker.json'
|
||||
temporary = worker.with_suffix('.bootstrap.tmp')
|
||||
temporary.write_text(json.dumps(identity), encoding='utf-8')
|
||||
temporary.replace(worker)
|
||||
try:
|
||||
journal = read(directory / 'journal.json', {})
|
||||
if journal.get('phase') == 'installing':
|
||||
restore_program_before_import(root, directory, journal)
|
||||
elif journal.get('phase') == 'stopping':
|
||||
stop_owned_services(root)
|
||||
else:
|
||||
raise RuntimeError('更新恢复阶段无效,未替换任何程序文件。')
|
||||
# This import occurs only after restoring a complete, verified old
|
||||
# program set; no mutable application module was imported above.
|
||||
path = safe(root, 'deploy/offline/update_manager.py')
|
||||
spec = importlib.util.spec_from_file_location('quality_recovered_updater', path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module.recover_locked(root)
|
||||
finally:
|
||||
if read(worker, {}) == identity:
|
||||
worker.unlink(missing_ok=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
print(json.dumps(recover(), ensure_ascii=False))
|
||||
except Exception as error:
|
||||
print(json.dumps({'error': str(error) if isinstance(error, RuntimeError) else '更新恢复未完成,请保留本机备份并联系管理员。'}, ensure_ascii=False))
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,328 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Real, isolated MySQL application-update acceptance on the macOS build host.
|
||||
|
||||
Never reads the source deployment configuration or connects to its MySQL port.
|
||||
The test copies the application and creates its own database server and ports.
|
||||
Only HTTP transport and the target-runtime verify action are adapted for the
|
||||
host; signature verification, backup, file replacement, startup and rollback
|
||||
are the production updater implementation.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
from datetime import datetime, timezone
|
||||
import hashlib
|
||||
import http.cookiejar
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import traceback
|
||||
from unittest.mock import patch
|
||||
import urllib.request
|
||||
import zipfile
|
||||
|
||||
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
|
||||
from cryptography.hazmat.primitives.serialization import Encoding, PublicFormat
|
||||
|
||||
import update_manager as update
|
||||
|
||||
SOURCE = Path(__file__).resolve().parents[2]
|
||||
OPENER = urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||
|
||||
|
||||
def free_port():
|
||||
with socket.socket() as test:
|
||||
test.bind(('127.0.0.1', 0))
|
||||
return test.getsockname()[1]
|
||||
|
||||
|
||||
def backend_port():
|
||||
for port in (5285, 5284, 5283):
|
||||
try:
|
||||
with socket.socket() as test:
|
||||
test.bind(('127.0.0.1', port))
|
||||
return port
|
||||
except OSError:
|
||||
continue
|
||||
raise RuntimeError('No unused standalone test backend port is available; no existing instance was touched.')
|
||||
|
||||
|
||||
def query(root, sql, database=None):
|
||||
config = update.read(root / '.runtime/deployment.json')
|
||||
command = [str(root / 'runtimes/mysql/bin/mysql'), '--no-defaults', '--protocol=TCP', '--host=127.0.0.1',
|
||||
'--port=' + str(config['mysqlPort']), '--user=root', '--connect-timeout=5', '--batch', '--skip-column-names',
|
||||
'--default-character-set=utf8mb4']
|
||||
if database:
|
||||
command.append(database)
|
||||
env = os.environ.copy()
|
||||
env['MYSQL_PWD'] = config['rootPassword']
|
||||
result = subprocess.run(command, input=sql, encoding='utf-8', capture_output=True, env=env, timeout=60)
|
||||
if result.returncode:
|
||||
(root / '.runtime/update/test-sql-error.log').write_text(result.stderr)
|
||||
raise RuntimeError('SQL failed in the isolated test server; see its private test log.')
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def web(root, path='/'):
|
||||
config = update.read(root / '.runtime/deployment.json')
|
||||
with OPENER.open(f'http://127.0.0.1:{config["webPort"]}' + path, timeout=10) as response:
|
||||
return response.read()
|
||||
|
||||
|
||||
def login(root, username, password):
|
||||
config = update.read(root / '.runtime/deployment.json')
|
||||
base = f'http://127.0.0.1:{config["webPort"]}'
|
||||
cookies = http.cookiejar.CookieJar()
|
||||
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}), urllib.request.HTTPCookieProcessor(cookies))
|
||||
with opener.open(base + '/api/auth/csrf', timeout=10) as response:
|
||||
token = json.load(response)['token']
|
||||
request = urllib.request.Request(base + '/api/auth/login', data=json.dumps({'username': username, 'password': password}).encode(),
|
||||
headers={'Content-Type': 'application/json', 'X-XSRF-TOKEN': token})
|
||||
with opener.open(request, timeout=15) as response:
|
||||
assert response.status == 200
|
||||
with opener.open(base + '/api/auth/me', timeout=10) as response:
|
||||
assert response.status == 200
|
||||
|
||||
|
||||
def prepare(work):
|
||||
# MySQL's Unix socket has a 103-byte path limit on this host.
|
||||
root = Path(tempfile.mkdtemp(prefix='qc-upd-', dir='/tmp')).resolve()
|
||||
(work / 'isolated-root.txt').write_text(str(root) + '\n')
|
||||
for name in ('app/web', 'scripts', 'deploy/offline', 'runtimes/python/bin', '.runtime/update',
|
||||
'runtimedata/logs', 'runtimedata/sources', 'runtimedata/archive', 'backend/src/main/resources'):
|
||||
(root / name).mkdir(parents=True, exist_ok=True)
|
||||
for name in ('control.py', 'gateway.py', 'update_manager.py'):
|
||||
shutil.copy2(SOURCE / 'deploy/offline' / name, root / 'deploy/offline' / name)
|
||||
shutil.copy2(SOURCE / 'deploy/offline/runtime_platform.py', root / 'scripts/runtime_platform.py')
|
||||
# No AI processes run in this acceptance test; preserve supervisor cleanup's
|
||||
# normal subprocess path while keeping it confined to the disposable root.
|
||||
(root / 'scripts/nao_quality_runtime.py').write_text('"""Isolated test: no AI processes have been started."""\n')
|
||||
shutil.copy2(SOURCE / 'backend/target/quality-platform-backend-0.1.0-SNAPSHOT.jar', root / 'app/backend.jar')
|
||||
shutil.copy2(SOURCE / 'backend/src/main/resources/analysis-runtime-policy.json', root / 'backend/src/main/resources/analysis-runtime-policy.json')
|
||||
(root / 'app/web/index.html').write_text('<!doctype html><html><body><div id="root">isolated-update-v1</div></body></html>')
|
||||
(root / 'app/web/obsolete.js').write_text('/* obsolete test application asset */')
|
||||
(root / 'runtimes/java').symlink_to(SOURCE / '.runtime/jdk/Contents/Home', target_is_directory=True)
|
||||
(root / 'runtimes/mysql').symlink_to(SOURCE / '.runtime/mysql', target_is_directory=True)
|
||||
(root / 'runtimes/python/bin/python3').symlink_to(sys.executable)
|
||||
key = Ed25519PrivateKey.generate()
|
||||
(root / update.KEY).write_bytes(key.public_key().public_bytes(Encoding.PEM, PublicFormat.SubjectPublicKeyInfo))
|
||||
config = {'mysqlPort': free_port(), 'backendPort': backend_port(), 'webPort': free_port(), 'webBind': '127.0.0.1',
|
||||
'mysqlDatabase': 'quality_platform', 'archiveDirectory': 'runtimedata/archive',
|
||||
'mysqlPassword': secrets.token_urlsafe(32), 'rootPassword': secrets.token_urlsafe(32),
|
||||
'adminUsername': 'update-test-admin', 'adminPassword': secrets.token_urlsafe(24)}
|
||||
assert config['mysqlPort'] != 53306
|
||||
update.atomic(root / '.runtime/deployment.json', config)
|
||||
(root / '.runtime/deployment.json').chmod(0o600)
|
||||
private_marker = secrets.token_hex(20)
|
||||
update.atomic(root / '.runtime/analysis-api-settings.json', {'testPrivateMarker': private_marker})
|
||||
manifest = {'platform': 'darwin', 'architecture': 'arm64', 'updateVersion': 'host-initial', 'updateSequence': 1,
|
||||
'runtimes': [{'name': 'java', 'hostTestRuntime': True}, {'name': 'mysql', 'hostTestRuntime': True},
|
||||
{'name': 'python', 'version': sys.version.split()[0], 'hostTestRuntime': True}],
|
||||
'pythonWheels': [], 'files': []}
|
||||
for path in sorted(root.rglob('*')):
|
||||
if path.is_file() and not path.is_symlink() and not path.relative_to(root).as_posix().startswith(('.runtime/', 'runtimedata/', 'runtimes/')):
|
||||
manifest['files'].append({'path': path.relative_to(root).as_posix(), 'sha256': update.digest(path)})
|
||||
update.atomic(root / 'manifest.json', manifest)
|
||||
return root, key, config, private_marker
|
||||
|
||||
|
||||
def release_fixture(root, work, key, version, sequence, bad=False):
|
||||
folder = work / ('fixture-' + version)
|
||||
folder.mkdir()
|
||||
program = {}
|
||||
for entry in update.read(root / 'manifest.json')['files']:
|
||||
name = entry['path']
|
||||
if update.program_path(name) and name != 'app/web/obsolete.js':
|
||||
program[name] = root / name
|
||||
index = folder / 'index.html'
|
||||
index.write_text(f'<!doctype html><html><body><div id="root">isolated-update-{version}</div></body></html>')
|
||||
program['app/web/index.html'] = index
|
||||
if bad:
|
||||
bad_jar = folder / 'bad-backend.jar'
|
||||
# The real new Java process runs Flyway, changes data, then fails. This
|
||||
# proves rollback restores the dump, rather than merely restarting Java.
|
||||
migration = ("UPDATE platform_metadata SET metadata_value='FAILED-UPDATE-DATA' WHERE metadata_key='update_integration_sentinel';\n"
|
||||
"UPDATE app_user SET password_hash='FAILED-UPDATE-HASH' WHERE username='update-test-admin';\n"
|
||||
"CREATE TABLE isolated_update_failure_marker(id INT PRIMARY KEY);\n"
|
||||
"SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT='Intentional isolated updater integration failure';\n")
|
||||
with zipfile.ZipFile(root / 'app/backend.jar') as source, zipfile.ZipFile(bad_jar, 'w') as target:
|
||||
for member in source.infolist():
|
||||
target.writestr(member, source.read(member))
|
||||
target.writestr('BOOT-INF/classes/db/migration/V999__isolated_update_failure.sql', migration)
|
||||
program['app/backend.jar'] = bad_jar
|
||||
archive = folder / ('quality-update-' + version + '.zip')
|
||||
with zipfile.ZipFile(archive, 'w', zipfile.ZIP_DEFLATED, compresslevel=1) as target:
|
||||
for name, path in sorted(program.items()):
|
||||
target.write(path, name)
|
||||
release = {'schema': 1, 'platform': 'windows-x64', 'version': version, 'sequence': sequence,
|
||||
'runtimeFingerprint': update.runtime_fingerprint(update.read(root / 'manifest.json')),
|
||||
'archive': {'file': archive.name, 'size': archive.stat().st_size, 'sha256': update.digest(archive)},
|
||||
'files': [{'path': name, 'size': path.stat().st_size, 'sha256': update.digest(path)} for name, path in sorted(program.items())]}
|
||||
payload = update.canonical(release)
|
||||
envelope = {'payload': base64.b64encode(payload).decode(), 'signature': base64.b64encode(key.sign(payload)).decode()}
|
||||
(folder / 'latest.json').write_bytes(update.canonical(envelope))
|
||||
return release, envelope, archive
|
||||
|
||||
|
||||
def run_update(root, release, envelope, archive):
|
||||
real_controller = update.controller
|
||||
verify_checks = []
|
||||
|
||||
def host_controller(target, action, database_only=False, timeout=1100):
|
||||
assert target == root
|
||||
if action != 'verify':
|
||||
return real_controller(target, action, database_only, timeout)
|
||||
# Host adaptation: this ARM Mac cannot satisfy Windows/x64 runtime probes.
|
||||
# All installed program hashes and the native Java/MySQL executables are
|
||||
# still checked. Real full application startup is checked below by serve.
|
||||
for entry in update.read(root / 'manifest.json')['files']:
|
||||
assert update.digest(root / entry['path']) == entry['sha256'], entry['path']
|
||||
for command in ([root / 'runtimes/java/bin/java', '-version'],
|
||||
[root / 'runtimes/mysql/bin/mysqld', '--no-defaults', '--version']):
|
||||
result = subprocess.run([str(a) for a in command], capture_output=True, timeout=30)
|
||||
assert result.returncode == 0
|
||||
verify_checks.append(True)
|
||||
|
||||
def transport(url, target=None, expected_size=None, expected_hash=None):
|
||||
if url == update.LATEST and target is None:
|
||||
return update.canonical(envelope)
|
||||
assert url == update.BASE + archive.name and target is not None
|
||||
assert expected_size == archive.stat().st_size and expected_hash == update.digest(archive)
|
||||
with archive.open('rb') as reader, target.open('xb') as writer:
|
||||
shutil.copyfileobj(reader, writer)
|
||||
return b''
|
||||
|
||||
launcher = subprocess.Popen([sys.executable, '-c', 'pass'])
|
||||
launcher.wait(timeout=10)
|
||||
token = secrets.token_hex(16)
|
||||
update.atomic(root / '.runtime/update/request.json', {'token': token, 'created': time.time()})
|
||||
# Patch only this test process. Production Windows/source enablement remains
|
||||
# unchanged, including in the real backend processes started by control.py.
|
||||
with patch.object(update, 'enabled', return_value=True), patch.object(update, 'download', side_effect=transport), \
|
||||
patch.object(update, 'controller', side_effect=host_controller):
|
||||
update.worker(launcher.pid, token, root)
|
||||
assert verify_checks, update.read(root / '.runtime/update/status.json').get('message')
|
||||
return update.read(root / '.runtime/update/status.json')
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--work', type=Path, required=True)
|
||||
parser.add_argument('--expected-tables', type=int, default=83)
|
||||
args = parser.parse_args()
|
||||
work = args.work.resolve()
|
||||
work.mkdir(parents=True, exist_ok=False)
|
||||
root = None
|
||||
report = {'startedAt': datetime.now(timezone.utc).isoformat(), 'passed': False,
|
||||
'environment': 'macOS ARM host, disposable native MySQL + Java + Python runtimes',
|
||||
'productionServicesTouched': False, 'productionDatabasePortUsed': False,
|
||||
'networkTransport': 'local signed fixtures substituted only in test process',
|
||||
'targetRuntimeVerify': 'host adapter: all manifest hashes plus native Java/MySQL probes; actual backend restarts',
|
||||
'windowsNativeProcessDetachmentValidated': False, 'windowsNativeRuntimeCompatibilityValidated': False,
|
||||
'checks': {}}
|
||||
try:
|
||||
print('Preparing isolated update acceptance root.', flush=True)
|
||||
root, key, config, private_marker = prepare(work)
|
||||
report['isolatedRoot'] = str(root)
|
||||
report['ports'] = {k: config[k] for k in ('mysqlPort', 'backendPort', 'webPort')}
|
||||
print('Starting isolated MySQL and Java; applying real Flyway schema.', flush=True)
|
||||
process = update.controller(root, 'serve')
|
||||
update.wait_ready(root, process)
|
||||
database = config['mysqlDatabase']
|
||||
count = int(query(root, f"SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='{database}';"))
|
||||
assert count == args.expected_tables, f'Expected {args.expected_tables} tables; actual isolated schema has {count}.'
|
||||
report['checks']['tableCount'] = count
|
||||
sentinel = secrets.token_hex(24)
|
||||
query(root, f"INSERT INTO platform_metadata(metadata_key,metadata_value) VALUES ('update_integration_sentinel','{sentinel}');", database)
|
||||
account_hash = query(root, "SELECT password_hash FROM app_user WHERE username='update-test-admin';", database)
|
||||
assert account_hash.startswith('$2')
|
||||
login(root, config['adminUsername'], config['adminPassword'])
|
||||
report['checks']['initialLogin'] = True
|
||||
config['restoredFromMigration'] = 'isolated-update-migration-fixture'
|
||||
update.atomic(root / '.runtime/deployment.json', config)
|
||||
update.atomic(root / '.runtime/migration-receipt.json', {'migrationId': config['restoredFromMigration'],
|
||||
'database': database, 'migrationManifestSha256': 'a' * 64})
|
||||
before_pid = update.read(root / '.runtime/service.json')['supervisor']['pid']
|
||||
print('Applying signed success release with actual dump, file replacement and service restart.', flush=True)
|
||||
good = release_fixture(root, work, key, 'host-success', 2)
|
||||
result = run_update(root, *good)
|
||||
assert result['state'] == 'completed', result.get('message')
|
||||
assert update.read(root / 'manifest.json')['updateVersion'] == 'host-success'
|
||||
assert b'isolated-update-host-success' in web(root)
|
||||
assert not (root / 'app/web/obsolete.js').exists()
|
||||
assert update.read(root / '.runtime/service.json')['supervisor']['pid'] != before_pid
|
||||
assert query(root, "SELECT metadata_value FROM platform_metadata WHERE metadata_key='update_integration_sentinel';", database) == sentinel
|
||||
assert query(root, "SELECT password_hash FROM app_user WHERE username='update-test-admin';", database) == account_hash
|
||||
login(root, config['adminUsername'], config['adminPassword'])
|
||||
first_journal = update.read(root / '.runtime/update/journal.json')
|
||||
first_dump = root / '.runtime/update' / first_journal['backup'] / 'database.sql'
|
||||
assert first_dump.is_file() and first_dump.stat().st_size > 10000
|
||||
report['checks'].update(realMysqldump=True, successProgramSwitch=True, realServiceRestart=True,
|
||||
obsoleteProgramRemoved=True, successSentinelPreserved=True,
|
||||
successAccountHashPreserved=True, successLogin=True,
|
||||
successDumpBytes=first_dump.stat().st_size)
|
||||
print('Applying signed bad Java release: real Flyway mutates sentinel/hash then intentionally fails.', flush=True)
|
||||
bad = release_fixture(root, work, key, 'host-bad', 3, bad=True)
|
||||
result = run_update(root, *bad)
|
||||
assert result['state'] == 'failed' and '已恢复原版本' in result['message'], result
|
||||
restored = update.read(root / '.runtime/deployment.json')
|
||||
restored_db = restored['mysqlDatabase']
|
||||
assert restored_db != database and restored_db.startswith('quality_restore_')
|
||||
assert update.read(root / 'manifest.json')['updateVersion'] == 'host-success'
|
||||
assert b'isolated-update-host-success' in web(root)
|
||||
assert query(root, "SELECT metadata_value FROM platform_metadata WHERE metadata_key='update_integration_sentinel';", restored_db) == sentinel
|
||||
assert query(root, "SELECT password_hash FROM app_user WHERE username='update-test-admin';", restored_db) == account_hash
|
||||
assert int(query(root, f"SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='{restored_db}';")) == count
|
||||
assert query(root, "SELECT metadata_value FROM platform_metadata WHERE metadata_key='update_integration_sentinel';", database) == 'FAILED-UPDATE-DATA', 'Failed original sentinel did not retain the migration write.'
|
||||
assert query(root, "SELECT password_hash FROM app_user WHERE username='update-test-admin';", database) == 'FAILED-UPDATE-HASH', 'Failed original account hash did not retain the migration write.'
|
||||
assert query(root, "SELECT COUNT(*) FROM flyway_schema_history WHERE version='999' AND success=0;", database) == '1'
|
||||
assert query(root, "SELECT COUNT(*) FROM flyway_schema_history WHERE version='999';", restored_db) == '0'
|
||||
assert update.read(root / '.runtime/analysis-api-settings.json')['testPrivateMarker'] == private_marker
|
||||
assert restored['archiveDirectory'] == config['archiveDirectory']
|
||||
assert restored['rootPassword'] == config['rootPassword'] and restored['mysqlPassword'] == config['mysqlPassword']
|
||||
assert update.read(root / '.runtime/migration-receipt.json')['database'] == restored_db
|
||||
login(root, config['adminUsername'], config['adminPassword'])
|
||||
failed_journal = update.read(root / '.runtime/update/journal.json')
|
||||
assert failed_journal['phase'] == 'rolledBack'
|
||||
failed_dump = root / '.runtime/update' / failed_journal['backup'] / 'database.sql'
|
||||
report['checks'].update(badReleaseActuallyMutatedDatabase=True, failedOriginalDatabasePreserved=True,
|
||||
restoredIntoNewDatabase=True, restoredTableCount=count, rollbackSentinelRestored=True,
|
||||
rollbackAccountHashRestored=True, rollbackLogin=True, oldProgramRestarted=True,
|
||||
privateConfigurationPreserved=True, archiveDirectoryPreserved=True,
|
||||
migrationReceiptDatabaseAligned=True,
|
||||
failedFlywayHistoryPreserved=True, restoredFlywayHistoryClean=True,
|
||||
rollbackDumpBytes=failed_dump.stat().st_size,
|
||||
originalDatabase=database, rollbackDatabase=restored_db)
|
||||
report['passed'] = True
|
||||
print('Real MySQL success and failed-migration rollback acceptance passed.', flush=True)
|
||||
except Exception as error:
|
||||
report['error'] = str(error)
|
||||
(work / 'test-failure-trace.txt').write_text(traceback.format_exc())
|
||||
print('Isolated acceptance failed: ' + str(error), flush=True)
|
||||
finally:
|
||||
if root is not None:
|
||||
try:
|
||||
update.controller(root, 'stop', timeout=180)
|
||||
state = update.read(root / '.runtime/service.json', {})
|
||||
assert not update.live(state.get('supervisor', {}))
|
||||
report['isolatedServicesStopped'] = True
|
||||
except Exception as error:
|
||||
report['isolatedServicesStopped'] = False
|
||||
report['cleanupError'] = str(error)
|
||||
report['passed'] = False
|
||||
report['finishedAt'] = datetime.now(timezone.utc).isoformat()
|
||||
(work / 'report.json').write_text(json.dumps(report, ensure_ascii=False, indent=2) + '\n')
|
||||
print(json.dumps(report, ensure_ascii=False), flush=True)
|
||||
return 0 if report['passed'] else 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,646 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Signed Windows standalone application updates. No caller-supplied URLs or paths.
|
||||
|
||||
The release envelope signs the *decoded payload bytes* with Ed25519. Runtime
|
||||
identity is SHA256(canonical({runtimes, pythonWheels})). Program files are a full
|
||||
snapshot; private data and runtimes are never update destinations.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import contextlib
|
||||
from datetime import datetime, timezone
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path, PurePosixPath
|
||||
import re
|
||||
import secrets
|
||||
import shutil
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import zipfile
|
||||
from urllib.request import ProxyHandler, HTTPRedirectHandler, build_opener
|
||||
|
||||
import psutil
|
||||
from cryptography.hazmat.primitives.serialization import load_pem_public_key
|
||||
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
BASE = 'https://quality.lyunlab.me/quality-updates/windows-x64/'
|
||||
LATEST = BASE + 'latest.json'
|
||||
KEY = 'deploy/offline/update-public-key.pem'
|
||||
MAX_MANIFEST = 16 * 1024 * 1024
|
||||
MAX_ARCHIVE = 2 * 1024**3
|
||||
MAX_EXPANDED = 4 * 1024**3
|
||||
ACTIVE = {'queued', 'downloading', 'backingUp', 'installing', 'restarting', 'rollingBack'}
|
||||
SHA = re.compile(r'[a-f0-9]{64}')
|
||||
|
||||
|
||||
class UpdateError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class BusyUpdate(UpdateError):
|
||||
pass
|
||||
|
||||
|
||||
def canonical(value):
|
||||
return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(',', ':')).encode('utf-8')
|
||||
|
||||
|
||||
def unique_object(pairs):
|
||||
value = {}
|
||||
for key, item in pairs:
|
||||
if key in value:
|
||||
raise UpdateError('更新清单包含重复字段。')
|
||||
value[key] = item
|
||||
return value
|
||||
|
||||
|
||||
def decode(raw):
|
||||
return json.loads(raw, object_pairs_hook=unique_object)
|
||||
|
||||
|
||||
def digest(path):
|
||||
with path.open('rb') as stream:
|
||||
return hashlib.file_digest(stream, 'sha256').hexdigest()
|
||||
|
||||
|
||||
def atomic(path, value):
|
||||
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
temporary = path.with_name(path.name + '.' + secrets.token_hex(6) + '.tmp')
|
||||
with temporary.open('xb') as stream:
|
||||
stream.write(canonical(value))
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
temporary.replace(path)
|
||||
|
||||
|
||||
def copy_atomic(source, target):
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = target.with_name(target.name + '.update-' + secrets.token_hex(6))
|
||||
try:
|
||||
with source.open('rb') as reader, temporary.open('xb') as writer:
|
||||
shutil.copyfileobj(reader, writer)
|
||||
writer.flush()
|
||||
os.fsync(writer.fileno())
|
||||
shutil.copystat(source, temporary)
|
||||
temporary.replace(target)
|
||||
finally:
|
||||
temporary.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def read(path, default=None):
|
||||
return decode(path.read_bytes()) if path.exists() else default
|
||||
|
||||
|
||||
def runtime_fingerprint(manifest):
|
||||
return hashlib.sha256(canonical({key: manifest.get(key, []) for key in ('runtimes', 'pythonWheels')})).hexdigest()
|
||||
|
||||
|
||||
def safe_path(name):
|
||||
if (not isinstance(name, str) or not name or len(name) > 230 or '\\' in name
|
||||
or ':' in name or '\x00' in name or name.startswith('/')
|
||||
or any(ord(c) < 32 for c in name)):
|
||||
raise UpdateError('更新文件路径无效。')
|
||||
parts = name.split('/')
|
||||
if any(p in ('', '.', '..') or p.endswith((' ', '.'))
|
||||
or re.fullmatch(r'(?i)(con|prn|aux|nul|com[1-9]|lpt[1-9])(?:\..*)?', p) for p in parts):
|
||||
raise UpdateError('更新文件路径无效。')
|
||||
return PurePosixPath(name)
|
||||
|
||||
|
||||
def program_path(name):
|
||||
path = safe_path(name)
|
||||
if name == KEY or name.startswith('deploy/bootstrap/'):
|
||||
return False # immutable trust anchor, even in a correctly signed release
|
||||
return (name == 'app/backend.jar' or name.startswith('app/web/')
|
||||
or name == 'backend/src/main/resources/analysis-runtime-policy.json'
|
||||
or name.startswith('scripts/') and path.suffix == '.py'
|
||||
or name.startswith('ai/nao-quality/')
|
||||
or name.startswith('deploy/offline/') and path.suffix in ('.py', '.ps1', '.md'))
|
||||
|
||||
|
||||
def destination(root, name):
|
||||
safe_path(name)
|
||||
target = root.joinpath(*name.split('/'))
|
||||
for path in (target, *target.parents):
|
||||
if path == root:
|
||||
break
|
||||
if path.is_symlink():
|
||||
raise UpdateError('更新目录包含符号链接,已停止更新。')
|
||||
if os.name == 'nt' and path.exists() and getattr(path.lstat(), 'st_file_attributes', 0) & stat.FILE_ATTRIBUTE_REPARSE_POINT:
|
||||
raise UpdateError('更新目录包含重解析点,已停止更新。')
|
||||
if not target.resolve().is_relative_to(root.resolve()):
|
||||
raise UpdateError('更新文件超出程序目录。')
|
||||
return target
|
||||
|
||||
|
||||
def entry_map(entries, program_only=False):
|
||||
if not isinstance(entries, list) or len(entries) > 50000:
|
||||
raise UpdateError('更新文件清单无效。')
|
||||
result, folded = {}, set()
|
||||
for entry in entries:
|
||||
if not isinstance(entry, dict):
|
||||
raise UpdateError('更新文件清单无效。')
|
||||
name = entry.get('path')
|
||||
safe_path(name)
|
||||
if name.casefold() in folded:
|
||||
raise UpdateError('更新文件路径重复。')
|
||||
folded.add(name.casefold())
|
||||
if program_only and not program_path(name):
|
||||
raise UpdateError('更新包含不允许替换的文件。')
|
||||
if entry.get('symlink') is not None or not SHA.fullmatch(str(entry.get('sha256', ''))):
|
||||
raise UpdateError('更新文件校验信息无效。')
|
||||
result[name] = entry
|
||||
return result
|
||||
|
||||
|
||||
def validate_release(envelope, root=ROOT):
|
||||
try:
|
||||
raw = base64.b64decode(envelope['payload'], validate=True)
|
||||
signature = base64.b64decode(envelope['signature'], validate=True)
|
||||
key = load_pem_public_key(destination(root, KEY).read_bytes())
|
||||
if not isinstance(key, Ed25519PublicKey):
|
||||
raise ValueError('key type')
|
||||
key.verify(signature, raw)
|
||||
release = decode(raw)
|
||||
except Exception as error:
|
||||
raise UpdateError('更新签名验证失败。') from error
|
||||
current = read(root / 'manifest.json')
|
||||
if release.get('schema') != 1 or release.get('platform') != 'windows-x64':
|
||||
raise UpdateError('更新包不适用于此系统。')
|
||||
if not isinstance(release.get('version'), str) or not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9._-]{0,79}', release['version']):
|
||||
raise UpdateError('更新版本无效。')
|
||||
if type(release.get('sequence')) is not int or release['sequence'] <= 0:
|
||||
raise UpdateError('更新版本序号无效。')
|
||||
if release.get('runtimeFingerprint') != runtime_fingerprint(current):
|
||||
raise UpdateError('运行环境版本不同,请使用完整安装包更新。')
|
||||
archive = release.get('archive', {})
|
||||
if (not re.fullmatch(r'quality-update-[A-Za-z0-9][A-Za-z0-9._-]{0,100}\.zip', str(archive.get('file', '')))
|
||||
or type(archive.get('size')) is not int or not 0 < archive['size'] <= MAX_ARCHIVE
|
||||
or not SHA.fullmatch(str(archive.get('sha256', '')))):
|
||||
raise UpdateError('更新下载信息无效。')
|
||||
files = entry_map(release.get('files'), True)
|
||||
if not files or any(type(e.get('size')) is not int or e['size'] < 0 for e in files.values()) or sum(e['size'] for e in files.values()) > MAX_EXPANDED:
|
||||
raise UpdateError('更新文件大小无效。')
|
||||
if 'bundleManifest' in release:
|
||||
raise UpdateError('更新清单不能包含部署数据。')
|
||||
old_files = entry_map(current.get('files'))
|
||||
required = {'app/backend.jar', 'deploy/offline/control.py', 'deploy/offline/update_manager.py',
|
||||
'backend/src/main/resources/analysis-runtime-policy.json'}
|
||||
if not required.issubset(files) or not any(name.startswith('app/web/') for name in files):
|
||||
raise UpdateError('更新缺少必要程序文件。')
|
||||
if KEY not in old_files or digest(destination(root, KEY)) != old_files[KEY]['sha256']:
|
||||
raise UpdateError('更新信任公钥校验失败。')
|
||||
for name in set(old_files) | set(files):
|
||||
destination(root, name)
|
||||
return release
|
||||
|
||||
|
||||
class NoRedirect(HTTPRedirectHandler):
|
||||
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
||||
raise UpdateError('更新服务器返回了不允许的跳转。')
|
||||
|
||||
|
||||
def download(url, target=None, expected_size=None, expected_hash=None):
|
||||
if url != LATEST and not (url.startswith(BASE) and re.fullmatch(r'quality-update-[A-Za-z0-9][A-Za-z0-9._-]{0,100}\.zip', url[len(BASE):])):
|
||||
raise UpdateError('更新下载地址无效。')
|
||||
limit = expected_size if expected_size is not None else MAX_MANIFEST
|
||||
hasher, count, chunks = hashlib.sha256(), 0, []
|
||||
with build_opener(ProxyHandler({}), NoRedirect()).open(url, timeout=30) as response:
|
||||
if response.geturl() != url or response.status != 200:
|
||||
raise UpdateError('更新服务器响应无效。')
|
||||
with target.open('xb') if target is not None else contextlib.nullcontext() as stream:
|
||||
while block := response.read(1024 * 1024):
|
||||
count += len(block)
|
||||
if count > limit:
|
||||
raise UpdateError('更新下载大小不符合清单。')
|
||||
hasher.update(block)
|
||||
if stream is not None:
|
||||
stream.write(block)
|
||||
else:
|
||||
chunks.append(block)
|
||||
if expected_size is not None and (count != expected_size or hasher.hexdigest() != expected_hash):
|
||||
raise UpdateError('更新下载校验失败。')
|
||||
return b''.join(chunks)
|
||||
|
||||
|
||||
def unpack(archive, stage, release):
|
||||
entries = {e['path']: e for e in release['files']}
|
||||
with zipfile.ZipFile(archive) as source:
|
||||
seen = set()
|
||||
for member in source.infolist():
|
||||
name = member.filename
|
||||
if name not in entries or name in seen or member.is_dir() or member.flag_bits & 1:
|
||||
raise UpdateError('更新压缩包文件清单不符。')
|
||||
seen.add(name)
|
||||
if stat.S_ISLNK(member.external_attr >> 16) or member.file_size != entries[name]['size']:
|
||||
raise UpdateError('更新压缩包文件类型或大小不符。')
|
||||
target = destination(stage, name)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
count = 0
|
||||
with source.open(member) as reader, target.open('xb') as writer:
|
||||
while block := reader.read(1024 * 1024):
|
||||
count += len(block)
|
||||
if count > entries[name]['size']:
|
||||
raise UpdateError('更新解压大小不符。')
|
||||
writer.write(block)
|
||||
if digest(target) != entries[name]['sha256']:
|
||||
raise UpdateError('更新解压校验失败。')
|
||||
if seen != set(entries):
|
||||
raise UpdateError('更新压缩包缺少文件。')
|
||||
|
||||
|
||||
def enabled(root=ROOT):
|
||||
return os.name == 'nt' and os.environ.get('QUALITY_UPDATE_ENABLED') == 'true' and (root / 'runtimes/python/python.exe').is_file() and (root / 'manifest.json').is_file()
|
||||
|
||||
|
||||
def status(root=ROOT):
|
||||
manifest = read(root / 'manifest.json', {})
|
||||
value = read(root / '.runtime/update/status.json', {})
|
||||
allowed = {'state', 'message', 'checkedAt', 'version', 'available', 'startedAt', 'finishedAt'}
|
||||
result = {k: v for k, v in value.items() if k in allowed}
|
||||
result.update(enabled=enabled(root), currentVersion=manifest.get('updateVersion', manifest.get('builtAtUtc', '当前版本')))
|
||||
result.setdefault('state', 'idle')
|
||||
if result['state'] in ACTIVE and not live(read(root / '.runtime/update/worker.json', {})):
|
||||
pending = read(root / '.runtime/update/request.json', {})
|
||||
journal = read(root / '.runtime/update/journal.json', {})
|
||||
if time.time() - pending.get('created', 0) > 120:
|
||||
result.update(state='failed', available=False, message='更新进程已中断,请重新运行软件启动入口恢复。'
|
||||
if journal.get('phase') not in (None, 'complete', 'rolledBack') else '更新进程已结束,请重新检查更新。')
|
||||
if not result['enabled']:
|
||||
result['message'] = '此部署不支持软件内更新。'
|
||||
return result
|
||||
|
||||
|
||||
def write_status(state, message, root=ROOT, **extra):
|
||||
destination(root, '.runtime/update/status.json')
|
||||
value = read(root / '.runtime/update/status.json', {})
|
||||
value.update(extra, state=state, message=message)
|
||||
atomic(root / '.runtime/update/status.json', value)
|
||||
return status(root)
|
||||
|
||||
|
||||
def check(root=ROOT):
|
||||
release = validate_release(decode(download(LATEST)), root)
|
||||
available = release['sequence'] > read(root / 'manifest.json').get('updateSequence', 0)
|
||||
atomic(root / '.runtime/update/checked.json', release)
|
||||
return write_status('available' if available else 'current', '发现新版本。' if available else '已是最新版本。', root,
|
||||
version=release['version'], available=available, checkedAt=datetime.now(timezone.utc).isoformat())
|
||||
|
||||
|
||||
def process_options():
|
||||
return {'creationflags': subprocess.DETACHED_PROCESS | subprocess.CREATE_NEW_PROCESS_GROUP | subprocess.CREATE_BREAKAWAY_FROM_JOB} if os.name == 'nt' else {'start_new_session': True}
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def exclusive_update(root=ROOT):
|
||||
directory = root / '.runtime/update'
|
||||
destination(root, '.runtime/update')
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
if live(read(directory / 'worker.json', {})):
|
||||
raise BusyUpdate('已有更新或恢复正在进行。')
|
||||
sys.path.insert(0, str(root / 'scripts'))
|
||||
from runtime_platform import fcntl
|
||||
with (directory / 'worker.lock').open('a+b') as lock:
|
||||
try:
|
||||
fcntl.flock(lock.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except (BlockingIOError, OSError):
|
||||
raise BusyUpdate('已有更新或恢复正在进行。') from None
|
||||
identity = {'pid': os.getpid(), 'created': psutil.Process().create_time()}
|
||||
atomic(directory / 'worker.json', identity)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
if read(directory / 'worker.json', {}) == identity:
|
||||
(directory / 'worker.json').unlink(missing_ok=True)
|
||||
|
||||
|
||||
def launch(root=ROOT):
|
||||
with exclusive_update(root):
|
||||
return launch_locked(root)
|
||||
|
||||
|
||||
def launch_locked(root=ROOT):
|
||||
directory = root / '.runtime/update'
|
||||
destination(root, '.runtime/update')
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
# This short launcher MUST exit before its worker may stop Java. The worker
|
||||
# verifies launcher disappearance; DETACHED alone does not remove ancestry.
|
||||
pending = read(directory / 'request.json', {})
|
||||
if pending and time.time() - pending.get('created', 0) < 120:
|
||||
raise BusyUpdate('已有更新正在启动。')
|
||||
check(root) # refetch and verify; do not trust browser or cached JSON
|
||||
if not status(root).get('available'):
|
||||
return status(root)
|
||||
token = secrets.token_hex(16)
|
||||
atomic(directory / 'request.json', {'token': token, 'created': time.time()})
|
||||
write_status('queued', '更新已开始,完成后将自动重新连接。', root, startedAt=datetime.now(timezone.utc).isoformat(), available=False)
|
||||
try:
|
||||
with (directory / 'worker.log').open('ab') as log:
|
||||
subprocess.Popen([sys.executable, str(Path(__file__).resolve()), 'worker', '--launcher', str(os.getpid()), '--token', token],
|
||||
cwd=root, env=os.environ.copy(), stdin=subprocess.DEVNULL, stdout=log, stderr=log, **process_options())
|
||||
except Exception:
|
||||
(directory / 'request.json').unlink(missing_ok=True)
|
||||
raise UpdateError('独立更新进程无法启动,服务仍在运行。') from None
|
||||
return status(root)
|
||||
|
||||
|
||||
def live(info):
|
||||
try:
|
||||
process = psutil.Process(info['pid'])
|
||||
return process.is_running() and process.status() != psutil.STATUS_ZOMBIE and abs(process.create_time() - info['created']) < .01
|
||||
except (psutil.Error, KeyError, TypeError):
|
||||
return False
|
||||
|
||||
|
||||
def controller(root, action, database_only=False, timeout=1100):
|
||||
python = root / 'runtimes/python' / ('python.exe' if os.name == 'nt' else 'bin/python3')
|
||||
command = [str(python), str(root / 'deploy/offline/control.py'), action]
|
||||
if database_only:
|
||||
command += ['--database-only']
|
||||
if action == 'serve':
|
||||
with (root / '.runtime/update/service.log').open('ab') as log:
|
||||
return subprocess.Popen(command, cwd=root, env=os.environ.copy(), stdin=subprocess.DEVNULL, stdout=log, stderr=log, **process_options())
|
||||
result = subprocess.run(command, cwd=root, env=os.environ.copy(), stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout)
|
||||
if result.returncode:
|
||||
raise UpdateError('服务启停失败,请查看本机更新日志。')
|
||||
return result
|
||||
|
||||
|
||||
def wait_ready(root, process, database_only=False):
|
||||
deadline = time.monotonic() + (180 if database_only else 1000)
|
||||
while time.monotonic() < deadline:
|
||||
state = read(root / '.runtime/service.json', {})
|
||||
if state.get('ready') and state.get('databaseOnly', False) == database_only and state.get('supervisor', {}).get('pid') == process.pid:
|
||||
return
|
||||
if process.poll() is not None:
|
||||
raise UpdateError('更新后服务启动失败,正在恢复旧版本。')
|
||||
time.sleep(.5)
|
||||
raise UpdateError('更新后服务启动超时,正在恢复旧版本。')
|
||||
|
||||
|
||||
def mysql(root, config, binary='mysql', database=None, stdin=None, stdout=subprocess.PIPE):
|
||||
exe = '.exe' if os.name == 'nt' else ''
|
||||
command = [str(root / 'runtimes/mysql/bin' / (binary + exe)), '--no-defaults', '--protocol=TCP', '--host=127.0.0.1',
|
||||
'--port=' + str(config['mysqlPort']), '--user=root', '--default-character-set=utf8mb4']
|
||||
if binary == 'mysqldump':
|
||||
command += ['--single-transaction', '--routines', '--events', '--triggers', '--hex-blob', '--no-tablespaces', '--set-gtid-purged=OFF', config.get('mysqlDatabase', 'quality_platform')]
|
||||
elif database:
|
||||
command.append(database)
|
||||
if binary == 'mysql':
|
||||
command.append('--connect-timeout=10')
|
||||
env = os.environ.copy()
|
||||
env['MYSQL_PWD'] = config['rootPassword']
|
||||
result = subprocess.run(command, cwd=root, env=env, stdin=stdin, stdout=stdout, stderr=subprocess.PIPE, timeout=3600)
|
||||
if result.returncode:
|
||||
error_log = destination(root, '.runtime/update/database-error.log')
|
||||
with error_log.open('ab') as log:
|
||||
log.write((binary + ' failed\n').encode() + result.stderr[-16384:] + b'\n')
|
||||
raise UpdateError('数据库备份或恢复失败,已保留本机备份和原数据库。')
|
||||
|
||||
|
||||
def sql(root, config, query):
|
||||
# stdin must be a real file on Windows; do not place passwords in arguments.
|
||||
path = root / '.runtime/update/query.sql'
|
||||
try:
|
||||
path.write_text(query, encoding='utf-8')
|
||||
with path.open('rb') as source:
|
||||
mysql(root, config, stdin=source)
|
||||
finally:
|
||||
path.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def backup_program(root, backup, release):
|
||||
current = read(root / 'manifest.json')
|
||||
names = sorted({e['path'] for e in current['files'] if program_path(e['path'])} | {e['path'] for e in release['files']})
|
||||
present = []
|
||||
for name in names:
|
||||
source = destination(root, name)
|
||||
if source.exists():
|
||||
target = destination(backup / 'program', name)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
copy_atomic(source, target)
|
||||
present.append(name)
|
||||
copy_atomic(root / 'manifest.json', backup / 'manifest.json')
|
||||
# All private configuration (AI keys, deployment credentials and bootstrap
|
||||
# flags) is backed up locally, but never becomes a release payload.
|
||||
shutil.copytree(root / '.runtime', backup / 'private', ignore=shutil.ignore_patterns('update', '*.lock', '*.log', 'service.json', 'stop-request.json'), symlinks=False)
|
||||
for path in (backup / 'private').rglob('*'):
|
||||
if path.is_file():
|
||||
with path.open('rb') as stream:
|
||||
os.fsync(stream.fileno())
|
||||
atomic(backup / 'program.json', {'all': names, 'present': present})
|
||||
|
||||
|
||||
def replace_program(root, stage, backup, release):
|
||||
current = read(root / 'manifest.json')
|
||||
names = read(backup / 'program.json')['all']
|
||||
wanted = {e['path'] for e in release['files']}
|
||||
for name in names:
|
||||
target = destination(root, name)
|
||||
if name in wanted:
|
||||
copy_atomic(stage / name, target)
|
||||
else:
|
||||
target.unlink(missing_ok=True)
|
||||
current['files'] = ([e for e in current['files'] if not program_path(e['path'])]
|
||||
+ [{'path': e['path'], 'sha256': e['sha256']} for e in release['files']])
|
||||
current.update(updateSequence=release['sequence'], updateVersion=release['version'])
|
||||
atomic(root / 'manifest.json', current)
|
||||
|
||||
|
||||
def restore_program(root, backup):
|
||||
names = read(backup / 'program.json')
|
||||
for name in names['all']:
|
||||
target = destination(root, name)
|
||||
if name in names['present']:
|
||||
copy_atomic(backup / 'program' / name, target)
|
||||
else:
|
||||
target.unlink(missing_ok=True)
|
||||
copy_atomic(backup / 'manifest.json', destination(root, 'manifest.json'))
|
||||
for source in (backup / 'private').rglob('*'):
|
||||
if source.is_file():
|
||||
relative = source.relative_to(backup / 'private').as_posix()
|
||||
copy_atomic(source, destination(root, '.runtime/' + relative))
|
||||
|
||||
|
||||
def rollback(root, backup, database_backed_up):
|
||||
write_status('rollingBack', '更新未完成,正在恢复原版本。', root)
|
||||
controller(root, 'stop', timeout=180)
|
||||
restore_program(root, backup)
|
||||
if database_backed_up:
|
||||
config = read(root / '.runtime/deployment.json')
|
||||
process = controller(root, 'serve', True)
|
||||
wait_ready(root, process, True)
|
||||
database = 'quality_restore_' + secrets.token_hex(12)
|
||||
sql(root, config, f'CREATE DATABASE `{database}` CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;')
|
||||
with (backup / 'database.sql').open('rb') as source:
|
||||
mysql(root, config, database=database, stdin=source)
|
||||
sql(root, config, f"GRANT ALL PRIVILEGES ON `{database}`.* TO 'quality_app'@'127.0.0.1';")
|
||||
controller(root, 'stop', timeout=180)
|
||||
config['mysqlDatabase'] = database
|
||||
atomic(root / '.runtime/deployment.json', config)
|
||||
receipt_path = root / '.runtime/migration-receipt.json'
|
||||
receipt = read(receipt_path, {})
|
||||
if receipt and receipt.get('migrationId') == config.get('restoredFromMigration'):
|
||||
receipt['database'] = database
|
||||
atomic(receipt_path, receipt)
|
||||
process = controller(root, 'serve')
|
||||
wait_ready(root, process)
|
||||
|
||||
|
||||
def worker(launcher, token, root=ROOT):
|
||||
directory = root / '.runtime/update'
|
||||
for _ in range(300):
|
||||
if not psutil.pid_exists(launcher):
|
||||
break
|
||||
time.sleep(.1)
|
||||
else:
|
||||
raise UpdateError('更新启动器未退出,已安全停止。')
|
||||
if any('java' in p.name().lower() for p in psutil.Process().parents()):
|
||||
raise UpdateError('更新进程未独立启动,已安全停止。')
|
||||
if read(directory / 'request.json', {}).get('token') != token:
|
||||
raise UpdateError('更新请求已失效。')
|
||||
with exclusive_update(root):
|
||||
backup = directory / ('backup-' + datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ') + '-' + secrets.token_hex(4))
|
||||
stopped = backed_up = program_saved = changed = False
|
||||
try:
|
||||
envelope = decode(download(LATEST))
|
||||
release = validate_release(envelope, root)
|
||||
if release['sequence'] <= read(root / 'manifest.json').get('updateSequence', 0):
|
||||
raise UpdateError('此更新版本已安装。')
|
||||
backup.mkdir()
|
||||
archive, stage = backup / 'update.zip', backup / 'stage'
|
||||
expanded = sum(entry['size'] for entry in release['files'])
|
||||
# Keep enough headroom for extraction, old program files and a
|
||||
# logical dump; the dump itself must complete before replacement.
|
||||
db_size = sum(p.stat().st_size for p in (root / 'runtimedata/mysql').rglob('*') if p.is_file())
|
||||
old_program_size = sum((root / entry['path']).stat().st_size for entry in read(root / 'manifest.json')['files'] if program_path(entry['path']) and (root / entry['path']).is_file())
|
||||
private_size = sum(p.stat().st_size for p in (root / '.runtime').rglob('*') if p.is_file() and not p.is_relative_to(directory))
|
||||
if shutil.disk_usage(root).free < release['archive']['size'] + expanded + old_program_size + private_size + db_size * 3 + 512 * 1024**2:
|
||||
raise UpdateError('磁盘剩余空间不足,请释放空间后重试。')
|
||||
write_status('downloading', '正在下载并校验更新。', root)
|
||||
info = release['archive']
|
||||
download(BASE + info['file'], archive, info['size'], info['sha256'])
|
||||
unpack(archive, stage, release)
|
||||
# Revalidate immediately before any service or disk mutation.
|
||||
validate_release(envelope, root)
|
||||
write_status('backingUp', '正在备份,服务即将重新启动。', root)
|
||||
atomic(directory / 'journal.json', {'backup': backup.name, 'phase': 'stopping', 'databaseBackedUp': False})
|
||||
controller(root, 'stop', timeout=180)
|
||||
stopped = True
|
||||
backup_program(root, backup, release)
|
||||
program_saved = True
|
||||
config = read(root / '.runtime/deployment.json')
|
||||
database = controller(root, 'serve', True)
|
||||
wait_ready(root, database, True)
|
||||
with (backup / 'database.sql').open('xb') as output:
|
||||
mysql(root, config, 'mysqldump', stdout=output)
|
||||
output.flush()
|
||||
os.fsync(output.fileno())
|
||||
if (backup / 'database.sql').stat().st_size < 100:
|
||||
raise UpdateError('数据库备份为空,已停止更新。')
|
||||
backed_up = True
|
||||
controller(root, 'stop', timeout=180)
|
||||
# The actual dump can exceed the source's physical storage. Do not
|
||||
# replace anything unless a separate rollback database still fits.
|
||||
required_recovery = max(db_size, (backup / 'database.sql').stat().st_size * 3) + 512 * 1024**2
|
||||
if shutil.disk_usage(root).free < required_recovery:
|
||||
raise UpdateError('备份已完成,但剩余磁盘空间不足以安全回退,已停止更新。')
|
||||
atomic(directory / 'journal.json', {'backup': backup.name, 'phase': 'installing', 'databaseBackedUp': True,
|
||||
'backupManifestSha256': digest(backup / 'manifest.json'),
|
||||
'programInventorySha256': digest(backup / 'program.json')})
|
||||
write_status('installing', '正在安装更新。', root)
|
||||
changed = True
|
||||
replace_program(root, stage, backup, release)
|
||||
controller(root, 'verify', timeout=180)
|
||||
write_status('restarting', '正在重新启动,请稍候。', root)
|
||||
process = controller(root, 'serve')
|
||||
wait_ready(root, process)
|
||||
atomic(directory / 'journal.json', {'backup': backup.name, 'phase': 'complete', 'databaseBackedUp': True})
|
||||
write_status('completed', '更新完成。', root, version=release['version'], finishedAt=datetime.now(timezone.utc).isoformat(), available=False)
|
||||
except Exception as error:
|
||||
message = str(error) if isinstance(error, UpdateError) else '更新未完成,请查看本机更新日志。'
|
||||
if stopped:
|
||||
try:
|
||||
if program_saved and changed:
|
||||
rollback(root, backup, backed_up)
|
||||
else:
|
||||
controller(root, 'stop', timeout=180)
|
||||
process = controller(root, 'serve')
|
||||
wait_ready(root, process)
|
||||
atomic(directory / 'journal.json', {'backup': backup.name, 'phase': 'rolledBack', 'databaseBackedUp': backed_up})
|
||||
message += ' 已恢复原版本。'
|
||||
except Exception:
|
||||
message += ' 自动恢复未完成,请联系管理员使用本机更新备份恢复。'
|
||||
write_status('failed', message, root, available=False, finishedAt=datetime.now(timezone.utc).isoformat())
|
||||
finally:
|
||||
(directory / 'request.json').unlink(missing_ok=True)
|
||||
|
||||
|
||||
def recover(root=ROOT):
|
||||
with exclusive_update(root):
|
||||
return recover_locked(root)
|
||||
|
||||
|
||||
def recover_locked(root=ROOT):
|
||||
directory = root / '.runtime/update'
|
||||
journal = read(directory / 'journal.json', {})
|
||||
if journal.get('phase') in (None, 'complete', 'rolledBack'):
|
||||
return status(root)
|
||||
name = journal.get('backup', '')
|
||||
if not re.fullmatch(r'backup-[0-9]{8}T[0-9]{6}Z-[a-f0-9]{8}', name):
|
||||
raise UpdateError('更新恢复记录无效。')
|
||||
backup = directory / name
|
||||
if journal.get('phase') == 'installing':
|
||||
if (digest(backup / 'manifest.json') != journal.get('backupManifestSha256')
|
||||
or digest(backup / 'program.json') != journal.get('programInventorySha256')):
|
||||
raise UpdateError('更新恢复备份校验失败,已保留现场。')
|
||||
rollback(root, backup, journal.get('databaseBackedUp') is True)
|
||||
else:
|
||||
controller(root, 'stop', timeout=180)
|
||||
process = controller(root, 'serve')
|
||||
wait_ready(root, process)
|
||||
atomic(directory / 'journal.json', {**journal, 'phase': 'rolledBack'})
|
||||
return write_status('failed', '上次更新中断,已恢复原版本。', root, available=False)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action', choices=('status', 'check', 'launch', 'worker', 'recover'))
|
||||
parser.add_argument('--launcher', type=int)
|
||||
parser.add_argument('--token')
|
||||
args = parser.parse_args()
|
||||
if args.action == 'status':
|
||||
return status()
|
||||
if not enabled():
|
||||
raise UpdateError('此部署不支持软件内更新。')
|
||||
if args.action == 'worker':
|
||||
if not args.launcher or not re.fullmatch(r'[a-f0-9]{32}', args.token or ''):
|
||||
raise UpdateError('更新请求无效。')
|
||||
worker(args.launcher, args.token)
|
||||
return status()
|
||||
if args.action == 'recover':
|
||||
return recover()
|
||||
journal = read(ROOT / '.runtime/update/journal.json', {})
|
||||
if journal.get('phase') not in (None, 'complete', 'rolledBack'):
|
||||
raise UpdateError('上次更新中断,请重新运行软件启动入口恢复。')
|
||||
if live(read(ROOT / '.runtime/update/worker.json', {})):
|
||||
raise BusyUpdate('已有更新正在进行。')
|
||||
return check() if args.action == 'check' else launch()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
print(json.dumps(main(), ensure_ascii=False))
|
||||
except Exception as error:
|
||||
message = str(error) if isinstance(error, UpdateError) else '更新服务暂不可用,请稍后重试。'
|
||||
if enabled() and not isinstance(error, BusyUpdate) and sys.argv[1:2] != ['status'] and not live(read(ROOT / '.runtime/update/worker.json', {})):
|
||||
write_status('failed', message)
|
||||
print(json.dumps({'error': message}, ensure_ascii=False))
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,7 @@
|
||||
param([ValidateRange(1024,65535)][int]$Port = 5280)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
Write-Host '此兼容脚本不执行任何系统级准备操作,也不需要管理员权限。'
|
||||
Write-Host '运行库随离线包提供。请使用 windows-start.ps1 启动应用。'
|
||||
Write-Host "网页端口为 $Port;其他设备能否访问取决于 Windows 当前防火墙和网络策略。"
|
||||
exit 0
|
||||
@@ -0,0 +1,64 @@
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$root = [System.IO.Path]::GetFullPath((Split-Path (Split-Path $PSScriptRoot -Parent) -Parent))
|
||||
try {
|
||||
if (-not [Environment]::Is64BitOperatingSystem -or $env:PROCESSOR_ARCHITECTURE -eq 'ARM64') {
|
||||
throw '此离线包需要 Intel/AMD 64 位 Windows 电脑。'
|
||||
}
|
||||
|
||||
$manifestPath = Join-Path $root 'manifest.json'
|
||||
if (-not (Test-Path -LiteralPath $manifestPath -PathType Leaf)) {
|
||||
throw '未找到离线包清单 manifest.json。请解压完整离线包后再启动。'
|
||||
}
|
||||
$manifest = Get-Content -Raw -Encoding UTF8 $manifestPath | ConvertFrom-Json
|
||||
$maxRelativePath = 0
|
||||
foreach ($entry in $manifest.files) {
|
||||
if ($null -eq $entry.path -or $entry.path -isnot [string] -or [string]::IsNullOrWhiteSpace($entry.path)) {
|
||||
throw '离线包清单包含无效文件路径。请重新解压完整离线包。'
|
||||
}
|
||||
if ($entry.path.Length -gt $maxRelativePath) { $maxRelativePath = $entry.path.Length }
|
||||
}
|
||||
if ($maxRelativePath -eq 0) { throw '离线包清单没有文件路径。请重新解压完整离线包。' }
|
||||
$maxRootLength = 248 - $maxRelativePath
|
||||
if ($root.Length -gt $maxRootLength) {
|
||||
$recommended = if ($env:LOCALAPPDATA) { Join-Path $env:LOCALAPPDATA 'QualityCenter' } else { '当前用户可写的较短本地 NTFS 目录' }
|
||||
throw "部署目录路径过长:根路径和清单最长相对路径不能超过 248 个字符。请尝试 $recommended;若该路径仍过长,请选择更短的当前用户可写 NTFS 目录。"
|
||||
}
|
||||
$driveRoot = [System.IO.Path]::GetPathRoot($root)
|
||||
$drive = [System.IO.DriveInfo]::new($driveRoot)
|
||||
if ($drive.DriveType -ne [System.IO.DriveType]::Fixed -or $drive.DriveFormat -ne 'NTFS') {
|
||||
throw '请将离线包解压到本机固定 NTFS 磁盘上的当前用户可写目录。'
|
||||
}
|
||||
|
||||
$python = Join-Path $root 'runtimes\python\python.exe'
|
||||
if (-not (Test-Path -LiteralPath $python -PathType Leaf)) {
|
||||
throw '未找到随包提供的 Python 运行环境。请重新解压完整离线包。'
|
||||
}
|
||||
$env:PYTHONUTF8 = '1'
|
||||
$env:PYTHONIOENCODING = 'utf-8'
|
||||
$env:QUALITY_UPDATE_ENABLED = 'true'
|
||||
|
||||
# Recover an interrupted app-local update before reading mutable deployment
|
||||
# configuration or starting the local service processes.
|
||||
& $python (Join-Path $root 'deploy\bootstrap\update-recovery.py')
|
||||
if ($LASTEXITCODE -ne 0) { throw '更新恢复仍未完成。请保留本地更新备份后重试。' }
|
||||
|
||||
$port = 5280
|
||||
$config = Join-Path $root '.runtime\deployment.json'
|
||||
if (Test-Path -LiteralPath $config -PathType Leaf) {
|
||||
$port = [int](Get-Content -Raw -Encoding UTF8 $config | ConvertFrom-Json).webPort
|
||||
}
|
||||
if ($port -lt 1024 -or $port -gt 65535) { throw '网页端口配置无效。' }
|
||||
|
||||
if (Test-Path (Join-Path $root 'migration\manifest.json')) {
|
||||
& $python (Join-Path $root 'deploy\offline\restore_migration.py')
|
||||
if ($LASTEXITCODE -ne 0) { throw '数据恢复未完成。现有数据库已保留,请查看 runtimedata\logs。' }
|
||||
}
|
||||
|
||||
Write-Host '正在以当前用户启动本机应用进程;不会安装系统组件、创建 Windows 服务或更改防火墙规则。'
|
||||
Write-Host "网页服务使用内网监听,端口为 $port。其他设备能否访问取决于 Windows 当前防火墙和网络策略。"
|
||||
& $python (Join-Path $root 'deploy\offline\control.py') start --lan
|
||||
if ($LASTEXITCODE -ne 0) { throw '启动未完成。请查看 runtimedata\logs\supervisor.log。' }
|
||||
} catch {
|
||||
Write-Host $_ -ForegroundColor Red
|
||||
exit 1
|
||||
}
|
||||
Reference in New Issue
Block a user