新增:Linux Docker 部署与自动回收、备份及容量监测
This commit is contained in:
@@ -0,0 +1,12 @@
|
||||
# 此文件只放非敏感选项,真实密码由 manage.py init 生成。
|
||||
HTTP_BIND=127.0.0.1
|
||||
HTTP_PORT=5280
|
||||
QUALITY_IMAGE_TAG=local
|
||||
MYSQL_DATABASE=quality_platform
|
||||
MYSQL_USER=quality_app
|
||||
QUALITY_ADMIN_USERNAME=admin
|
||||
JAVA_MAX_HEAP=2048m
|
||||
# 从旧库迁移时必须核实旧版本会话的时区,不要猜测。
|
||||
MYSQL_LEGACY_TIMESTAMP_OFFSET_SECONDS=UNCONFIRMED
|
||||
QUALITY_ANALYSIS_PROTOCOL_V1_ENABLED=false
|
||||
QUALITY_ANALYSIS_AUTONOMOUS_ENABLED=false
|
||||
@@ -0,0 +1,52 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
FROM maven:3.9-eclipse-temurin-21 AS java-build
|
||||
WORKDIR /build
|
||||
COPY backend/pom.xml backend/pom.xml
|
||||
RUN --mount=type=cache,target=/root/.m2 mvn -B -f backend/pom.xml dependency:go-offline
|
||||
COPY backend/src backend/src
|
||||
# 默认执行不需要私有工作簿的测试;不把生产库传入构建。
|
||||
RUN --mount=type=cache,target=/root/.m2 mvn -B -f backend/pom.xml package
|
||||
|
||||
FROM python:3.11-slim-bookworm AS runtimes
|
||||
ARG TARGETARCH
|
||||
RUN test "$TARGETARCH" = amd64
|
||||
WORKDIR /build
|
||||
COPY deploy/docker/assets.lock.json deploy/docker/fetch_runtimes.py ./
|
||||
RUN python fetch_runtimes.py
|
||||
|
||||
FROM debian:bookworm-slim AS app
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
ca-certificates tzdata tini procps libaio1 libnuma1 libncurses6 libtirpc3 libstdc++6 libgomp1 \
|
||||
libatomic1 libcrypt1 libzstd1 liblz4-1 libx11-6 libxext6 libxi6 libxrender1 libxtst6 libasound2 fontconfig fonts-dejavu-core \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& groupadd -g 10001 quality && useradd -u 10001 -g quality -m quality
|
||||
COPY --from=runtimes /runtimes/jre /opt/java
|
||||
COPY --from=runtimes /runtimes/python /opt/python
|
||||
COPY --from=runtimes /runtimes/mysql /opt/mysql
|
||||
ENV JAVA_HOME=/opt/java PATH=/opt/nao/bin:/opt/java/bin:/opt/mysql/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
|
||||
PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 QUALITY_CONTAINER=1 TZ=Asia/Shanghai
|
||||
COPY deploy/docker/requirements-linux.lock deploy/macos/retention-requirements.txt /tmp/requirements/
|
||||
RUN /opt/python/bin/python3 -m venv /opt/nao \
|
||||
&& /opt/nao/bin/pip install --no-cache-dir --require-hashes -r /tmp/requirements/requirements-linux.lock -r /tmp/requirements/retention-requirements.txt \
|
||||
&& /opt/nao/bin/python -c "import nao_core, pymysql, psutil; import importlib.metadata as m; assert m.version('nao-core') == '0.3.26'" \
|
||||
&& /opt/java/bin/java -version && /opt/mysql/bin/mysqld --version \
|
||||
&& rm -rf /tmp/requirements
|
||||
WORKDIR /srv/quality
|
||||
COPY .dockerignore runtime.lock.json requirements-nao.in requirements-nao.lock ./
|
||||
COPY backend/pom.xml backend/pom.xml
|
||||
COPY backend/src backend/src
|
||||
COPY web/package.json web/pnpm-lock.yaml web/index.html web/tsconfig.json web/vite.config.ts web/pnpm-workspace.yaml web/
|
||||
COPY web/src web/src
|
||||
COPY web/public web/public
|
||||
COPY scripts scripts
|
||||
COPY ai ai
|
||||
COPY deploy/macos deploy/macos
|
||||
COPY deploy/docker deploy/docker
|
||||
COPY --from=java-build /build/backend/target/quality-platform-backend-0.1.0-SNAPSHOT.jar app/backend.jar
|
||||
RUN sha256sum app/backend.jar | cut -d ' ' -f 1 > app/backend.sha256 \
|
||||
&& mkdir -p .runtime output/backups output/logs runtimedata/archive \
|
||||
&& chown -R quality:quality .runtime output runtimedata
|
||||
USER 10001:10001
|
||||
EXPOSE 5281
|
||||
ENTRYPOINT ["/usr/bin/tini", "--", "/opt/nao/bin/python", "deploy/docker/app.py"]
|
||||
CMD ["serve"]
|
||||
@@ -0,0 +1,13 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
FROM node:22-bookworm-slim AS build
|
||||
RUN npm install --global pnpm@10.15.1
|
||||
WORKDIR /build
|
||||
COPY web/package.json web/pnpm-lock.yaml web/pnpm-workspace.yaml ./
|
||||
RUN --mount=type=cache,target=/root/.local/share/pnpm/store pnpm install --frozen-lockfile
|
||||
COPY web ./
|
||||
RUN pnpm build
|
||||
|
||||
FROM nginx:stable-alpine
|
||||
COPY deploy/docker/nginx.conf /etc/nginx/conf.d/default.conf
|
||||
COPY --from=build /build/dist /usr/share/nginx/html
|
||||
EXPOSE 8080
|
||||
@@ -0,0 +1,76 @@
|
||||
# Linux Docker 部署
|
||||
|
||||
本目录是项目源码的一部分,随 Gitea 仓库同步。它提供手动构建与运行入口,不需要 Gitea Actions、Runner 或自动发布流水线。构建需要联网获取公共依赖;已构建镜像在启动时不运行 Maven、pnpm、pip 或下载运行时。AI 调用仍需要可用的模型服务。
|
||||
|
||||
当前准备目标为 **Linux x86_64 / Docker Engine / Compose V2(建议 2.24 以上)**。应用运行时固定为 Java 21.0.13、Python 3.11.15、MySQL 8.4.13、nao-core 0.3.26。原生运行时校验和在 `assets.lock.json`,Python 依赖使用带哈希的锁文件。基础系统镜像按发行系列获取安全更新;正式发布时应记录构建提交和镜像 digest。ARM64 不在本版本支持范围内。
|
||||
|
||||
## 服务与数据
|
||||
|
||||
| 服务 | 内容 | 数据位置 |
|
||||
| --- | --- | --- |
|
||||
| mysql | MySQL,关闭 binary log,保留事务 redo/undo | `mysql_data` 命名卷 |
|
||||
| app | 后端、AI、自动回收、备份验证、容量监测 | `state`、`archives`、`backups`、`logs` 命名卷 |
|
||||
| web | 前端与 API 代理,支持流式响应 | 镜像内静态文件 |
|
||||
|
||||
只发布网页端口,默认 `127.0.0.1:5280`;MySQL 和后端没有宿主机端口映射。应用以 UID 10001 运行,镜像只读,不挂载 Docker socket。后端及所有维护子进程共享同一容器的进程空间,避免误判其他容器内的校验任务已经退出。不要同时启动第二个 app 副本访问同一组数据卷。
|
||||
|
||||
首次初始化后,密码存放于项目私有目录 `.runtime/docker/secrets/`,不是源码、镜像层或命令参数。Compose 从管理入口的环境变量读取密码,再作为 secret 文件交给容器。主机管理员仍能查看容器配置,需限制 Docker 管理权限。数据库初始化后不要直接重新生成密码;密码轮换必须同步修改数据库账户。
|
||||
|
||||
## 从仓库构建与启动
|
||||
|
||||
在项目根目录执行:
|
||||
|
||||
```sh
|
||||
python3 deploy/docker/manage.py init
|
||||
python3 deploy/docker/manage.py config
|
||||
python3 deploy/docker/manage.py build
|
||||
python3 deploy/docker/manage.py up
|
||||
python3 deploy/docker/manage.py status
|
||||
```
|
||||
|
||||
`init` 不覆盖已有设置或密码。非敏感设置位于 `.runtime/docker/config.env`;内网访问时可把 `HTTP_BIND` 改为服务器内网 IP,配合防火墙限制访问范围。管理员初始密码位于私有文件 `.runtime/docker/secrets/admin_password`,登录后及时更改。首次克隆为空系统,业务数据需单独授权导入。
|
||||
|
||||
启用服务器上的 Docker 开机启动后,已创建且未手动停止的容器通过 `restart: unless-stopped` 随 Docker 自动启动;MySQL 就绪后才启动应用,应用就绪后再启动网页。后端异常退出会使 app 退出并由 Docker 重启;维护进程退出由入口进程重新拉起。健康检查用于报告就绪状态,不会因一次查询超时强杀正在工作的数据库。
|
||||
|
||||
```sh
|
||||
sudo systemctl enable --now docker
|
||||
python3 deploy/docker/manage.py logs
|
||||
python3 deploy/docker/manage.py stop
|
||||
```
|
||||
|
||||
`stop` 保留所有数据卷,并表示主动停服;之后用 `up` 恢复。不要执行 `docker compose down -v`,它会删除数据库、归档和备份卷。操作系统关机时仍会停服;要摆脱测试 Mac 的关机和网络影响,必须将整套服务部署在持续开机联网的服务器上。
|
||||
|
||||
## 自动控制空间增长
|
||||
|
||||
复用 `deploy/macos/retention.py` 中的平台通用回收逻辑和 `scripts/` 内的保留模块;macOS 的 launchd 安装代码不会在 Linux 执行。
|
||||
|
||||
- 每 5 分钟检查旧版本、终态无引用构建、过期上传草稿和无引用旧来源修订。当前快照、候选基线、保存的报表及分析引用均保护;删除前重新确认当前版本与任务状态。
|
||||
- 导入、分析、排队、重试或未知状态任务存在时延期,锁冲突也延期,不终止任务。无引用对象连续 30 分钟后才进入回收;上传草稿先经过 24 小时有效期。
|
||||
- 发现新正式版本或距上次备份超过 24 小时,在空闲时生成候选备份。新候选通过独立、无网络 MySQL 实例恢复、表行数、业务汇总、来源归档及应用文件校验后,才替换上一份有效备份。失败时保留旧备份;创建期间会短暂同时存在新旧两份及恢复校验空间。
|
||||
- 测试数据库按所有者、进程身份、服务器 UUID 和标记登记。正常完成即清理;异常遗留超过 24 小时且所有者及子进程已退出后再处理。不按名称前缀批量删除未知数据库。新增数据库集成测试应使用 `OwnedTestDatabase` 或 `managed_test_resources.py`,并在 finally 中释放。
|
||||
- Java 日志按日期及每段 10MB 滚动,保留 7 天,归档总量上限 80MB,为活动文件留出空间。关闭的其他项目日志每类最多 100MiB、最长 7 天;活动文件受保护。Docker 每个容器日志使用 `local` 驱动、10m×10 段限制。MySQL 错误日志进入受限制的容器日志;binary log 关闭后不支持历史时间点恢复,恢复依赖最新有效快照。
|
||||
- 容量每 5 分钟采样,保留 30 天日报。业务数据、测试/校验临时空间分别达到 30GiB 提醒、50GiB 严重提醒;磁盘余量低于 50/20GiB、一天增长超过 2GiB 且超过 20%、连续维护失败也告警。告警写入容器日志及 `state` 卷的 `operations/alerts.jsonl`,同类 6 小时内去重。阈值在 `state` 卷的 `operations/policy.json`,不应为压低统计而删除业务数据。
|
||||
|
||||
Linux 的 MySQL 用量来自 `information_schema` 表和索引估算,不包含系统表空间、redo/undo 和 Docker 镜像缓存;宿主机还应使用磁盘监测与 `docker system df` 监测这些空间。容器内报告的可用空间取各业务卷的最小值;MySQL 卷所在磁盘也要由主机监控。此版本不提供 Linux 桌面弹窗或外部通知推送,容器日志可接入主机现有告警系统。
|
||||
|
||||
手动生成并验证一份备份:
|
||||
|
||||
```sh
|
||||
python3 deploy/docker/manage.py backup
|
||||
```
|
||||
|
||||
维护记录分别在 `state` 卷的 `retention/state.json`、`operations/monitor.json`。运行包或数据库结构改变时,回收会暂停;检查新版本兼容性后才执行 `python3 deploy/docker/manage.py review-retention`。此命令只更新兼容性标记,不取消任务保护或备份校验。
|
||||
|
||||
## 数据迁移与公网入口
|
||||
|
||||
当前测试 Mac 不会因本目录更新而切换运行包、迁移数据库或改变公网入口。源码中也不包含测试库、来源表格、现有备份和 API 密钥。
|
||||
|
||||
正式迁移采用逻辑备份及来源归档,不把 Mac 的 MySQL 数据目录直接复制到 Linux。先做 `backup.py create-and-verify`,保留原机;在新服务器的新数据库和独立目录恢复并重绑定归档绝对路径,核对用户、当前发布版本、月度汇总、归档摘要、登录和 AI,再切换域名。`backup.py restore --help` 提供新库/新目录的防覆盖恢复入口;它不是自动覆盖容器数据卷的导入命令。旧库的时间迁移参数必须从原部署核实。
|
||||
|
||||
长期入口建议为:域名 → 服务器 HTTPS 反向代理 → 本服务器 web → app → MySQL。公网只开放 443(按需开放 80 做跳转/证书),不再依赖测试 Mac 的 SSH 隧道。另一条路径是服务器应用连接托管 MySQL,另配对象存储及异地备份;该方案需要单独评估数据库权限、备份方式及来源路径,不能直接套用本地卷配置。
|
||||
|
||||
## 验收边界
|
||||
|
||||
本目录当前已准备源码与测试;没有 Linux Docker 主机时,不把静态检查或 Mac 上的测试称作容器启动成功。上主机后至少完成:构建三项服务、空库初始化、登录/导入/AI、后端与 MySQL 异常退出重启、主机重启、有效备份恢复、新旧数据对账、进行中任务保护、日志滚动及告警、HTTPS 与断开测试 Mac 后访问。容量规划还应为新旧备份和独立恢复校验预留余量。
|
||||
|
||||
参考:[Compose 启动顺序](https://docs.docker.com/compose/how-tos/startup-order/)、[Compose 服务与重启策略](https://docs.docker.com/reference/compose-file/services/)、[MySQL 官方镜像](https://hub.docker.com/_/mysql)、[Logback 日志保留](https://logback.qos.ch/manual/appenders-rolling.html)。
|
||||
@@ -0,0 +1,161 @@
|
||||
#!/usr/bin/env python3
|
||||
"""容器入口:固定运行包、私有配置、子进程守护和安全停止。"""
|
||||
import argparse
|
||||
import fcntl
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path[:0] = [str(ROOT / 'scripts'), str(ROOT / 'deploy/macos')]
|
||||
|
||||
|
||||
def secret(name):
|
||||
path = Path('/run/secrets') / name
|
||||
value = path.read_text().strip()
|
||||
if not re.fullmatch(r'[A-Za-z0-9_-]{24,128}', value):
|
||||
raise ValueError('Invalid secret: ' + name)
|
||||
return value
|
||||
|
||||
|
||||
def identifier(value):
|
||||
if not re.fullmatch(r'[A-Za-z][A-Za-z0-9_]{0,47}', value):
|
||||
raise ValueError('Invalid database identifier')
|
||||
return value
|
||||
|
||||
|
||||
def prepare(root=ROOT):
|
||||
import backup_retention as br
|
||||
root = Path(root)
|
||||
for relative in ('.runtime/retention', '.runtime/operations', '.runtime/nao-users',
|
||||
'.runtime/nao-trial', 'runtimedata/archive', 'output/backups', 'output/logs'):
|
||||
(root / relative).mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
for relative, destination in (('.runtime/mysql', '/opt/mysql'), ('.runtime/nao-trial/venv', '/opt/nao')):
|
||||
path = root / relative
|
||||
if path.is_symlink():
|
||||
if str(path.readlink()) != destination: raise ValueError('Unexpected runtime link')
|
||||
elif path.exists(): raise ValueError('Runtime path already occupied')
|
||||
else: path.symlink_to(destination, target_is_directory=True)
|
||||
value = {'mysqlHost': 'mysql', 'mysqlPort': 3306,
|
||||
'mysqlDatabase': identifier(os.environ.get('MYSQL_DATABASE', 'quality_platform')),
|
||||
'mysqlUser': identifier(os.environ.get('MYSQL_USER', 'quality_app')),
|
||||
'mysqlPassword': secret('mysql_password'), 'rootPassword': secret('mysql_root_password'),
|
||||
'adminUsername': os.environ.get('QUALITY_ADMIN_USERNAME', 'admin'),
|
||||
'adminPassword': secret('admin_password'), 'databaseReady': True, 'binaryLogEnabled': False,
|
||||
'legacyTimestampOffsetSeconds': os.environ.get('MYSQL_LEGACY_TIMESTAMP_OFFSET_SECONDS', 'UNCONFIRMED')}
|
||||
config = root / '.runtime/local-config.json'
|
||||
if config.exists():
|
||||
old = json.loads(config.read_text())
|
||||
# 更改持久数据库的密码需要专门轮换,不能在重启时默默写入不一致配置。
|
||||
for key in ('mysqlDatabase', 'mysqlUser', 'mysqlPassword', 'rootPassword'):
|
||||
if old.get(key) != value[key]: raise ValueError('Persisted database settings changed: ' + key)
|
||||
br.atomic(config, value)
|
||||
cnf = root / '.runtime/mysql-root.cnf'
|
||||
cnf.write_text('[client]\nuser=root\npassword=' + value['rootPassword'] + '\nhost=mysql\nport=3306\nprotocol=TCP\n')
|
||||
cnf.chmod(0o600)
|
||||
import operations
|
||||
policy = root / '.runtime/operations/policy.json'
|
||||
if not policy.exists(): br.atomic(policy, operations.DEFAULTS)
|
||||
return value
|
||||
|
||||
|
||||
def backend_environment(value):
|
||||
env = os.environ.copy()
|
||||
env.update({'MYSQL_URL': 'jdbc:mysql://mysql:3306/' + value['mysqlDatabase'] +
|
||||
'?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC&forceConnectionTimeZoneToSession=true&useSSL=false&allowPublicKeyRetrieval=true',
|
||||
'MYSQL_USER': value['mysqlUser'], 'MYSQL_PASSWORD': value['mysqlPassword'],
|
||||
'QUALITY_BOOTSTRAP_ADMIN_USERNAME': value['adminUsername'],
|
||||
'QUALITY_BOOTSTRAP_ADMIN_PASSWORD': value['adminPassword'],
|
||||
'SERVER_ADDRESS': '0.0.0.0', 'QUALITY_PROJECT_ROOT': str(ROOT),
|
||||
'QUALITY_ARCHIVE_ROOT': str(ROOT / 'runtimedata/archive'), 'QUALITY_ALLOWED_ROOTS': '',
|
||||
'LOGGING_CONFIG': str(ROOT / 'deploy/macos/logback-quality.xml'),
|
||||
'QUALITY_LOG_DIR': str(ROOT / 'output/logs'), 'QUALITY_CONTAINER': '1'})
|
||||
return env
|
||||
|
||||
|
||||
def ready():
|
||||
try:
|
||||
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||
with opener.open('http://127.0.0.1:5281/actuator/health', timeout=3) as response:
|
||||
return json.load(response).get('status') == 'UP'
|
||||
except (OSError, ValueError): return False
|
||||
|
||||
|
||||
def review_retention(first=False):
|
||||
import backup_retention as br
|
||||
from retention import Database
|
||||
path = ROOT / '.runtime/retention/config.json'
|
||||
if first and path.exists(): return
|
||||
with br.lock(ROOT, timeout=0):
|
||||
db = Database(ROOT)
|
||||
try:
|
||||
if db.busy(): raise RuntimeError('Application jobs are active; review deferred')
|
||||
accepted = br.accepted_backend(ROOT)
|
||||
if not accepted: raise RuntimeError('Pinned backend not found')
|
||||
br.atomic(path, {'schemaSignature': db.schema_signature, 'acceptedJarSha256': accepted[1],
|
||||
'retirementGraceSeconds': 1800, 'intervalSeconds': 300})
|
||||
finally: db.conn.close()
|
||||
|
||||
|
||||
def serve():
|
||||
import backup_retention as br
|
||||
(ROOT / '.runtime').mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
guard = (ROOT / '.runtime/container.lock').open('a')
|
||||
fcntl.flock(guard, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
value = prepare()
|
||||
br.accepted_backend(ROOT)
|
||||
heap = os.environ.get('JAVA_MAX_HEAP', '2048m')
|
||||
if not re.fullmatch(r'[1-9][0-9]*[mMgG]', heap): raise ValueError('Invalid Java heap size')
|
||||
env = backend_environment(value)
|
||||
children = {'backend': subprocess.Popen(['java', '-Xms128m', '-Xmx' + heap, '-jar', str(ROOT / 'app/backend.jar')],
|
||||
env=env, start_new_session=True)}
|
||||
stopping = False
|
||||
def stop(signum, frame):
|
||||
nonlocal stopping
|
||||
stopping = True
|
||||
signal.signal(signal.SIGTERM, stop); signal.signal(signal.SIGINT, stop)
|
||||
try:
|
||||
while not stopping:
|
||||
if children['backend'].poll() is not None: return 1
|
||||
if ready():
|
||||
try: review_retention(first=True)
|
||||
except Exception as error: print('retention-initialization: ' + type(error).__name__, flush=True)
|
||||
for role in ('retention', 'monitor'):
|
||||
if role not in children or children[role].poll() is not None:
|
||||
children[role] = subprocess.Popen([sys.executable, str(ROOT / 'deploy/docker/worker.py'), role],
|
||||
env=env, start_new_session=True)
|
||||
time.sleep(1)
|
||||
return 0
|
||||
finally:
|
||||
for process in children.values():
|
||||
if process.poll() is None:
|
||||
try: os.killpg(process.pid, signal.SIGTERM)
|
||||
except ProcessLookupError: pass
|
||||
deadline = time.monotonic() + 270
|
||||
for process in children.values():
|
||||
try: process.wait(timeout=max(.1, deadline - time.monotonic()))
|
||||
except subprocess.TimeoutExpired:
|
||||
os.killpg(process.pid, signal.SIGKILL); process.wait()
|
||||
guard.close()
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action', choices=['serve', 'health', 'review-retention'], default='serve', nargs='?')
|
||||
args = parser.parse_args()
|
||||
if args.action == 'health': return 0 if ready() else 1
|
||||
if args.action == 'review-retention': review_retention(); return 0
|
||||
return serve()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try: raise SystemExit(main())
|
||||
except Exception as error:
|
||||
print('Container startup failed: ' + type(error).__name__, file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,20 @@
|
||||
[
|
||||
{
|
||||
"name": "jre",
|
||||
"version": "21.0.13+9.0.LTS",
|
||||
"url": "https://github.com/adoptium/temurin21-binaries/releases/download/jdk-21.0.13%2B9/OpenJDK21U-jre_x64_linux_hotspot_21.0.13_9.tar.gz",
|
||||
"sha256": "a535a1df9c6284d9618872984e026ae0352d1f7e9e0605425c0e6114f47351dd"
|
||||
},
|
||||
{
|
||||
"name": "python",
|
||||
"version": "3.11.15",
|
||||
"url": "https://github.com/astral-sh/python-build-standalone/releases/download/20260623/cpython-3.11.15%2B20260623-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz",
|
||||
"sha256": "0604cd029b142dc223e131f17f5941c0c8d2d5074997c8178b515b19eea2a6c2"
|
||||
},
|
||||
{
|
||||
"name": "mysql",
|
||||
"version": "8.4.13",
|
||||
"url": "https://cdn.mysql.com/Downloads/MySQL-8.4/mysql-8.4.13-linux-glibc2.28-x86_64-minimal.tar.xz",
|
||||
"sha256": "5dfbb764f731df40ef9e4f5d69aed32bdac8ba1af20bd96b7d40a942d83d4ed2"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,108 @@
|
||||
name: quality-center
|
||||
|
||||
x-logging: &bounded-logging
|
||||
driver: local
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "10"
|
||||
|
||||
services:
|
||||
mysql:
|
||||
image: mysql:8.4.13
|
||||
platform: linux/amd64
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
MYSQL_DATABASE: ${MYSQL_DATABASE:-quality_platform}
|
||||
MYSQL_USER: ${MYSQL_USER:-quality_app}
|
||||
MYSQL_ROOT_HOST: "%"
|
||||
MYSQL_ROOT_PASSWORD_FILE: /run/secrets/mysql_root_password
|
||||
MYSQL_PASSWORD_FILE: /run/secrets/mysql_password
|
||||
TZ: Asia/Shanghai
|
||||
secrets: [mysql_root_password, mysql_password]
|
||||
command:
|
||||
- --skip-log-bin
|
||||
- --mysqlx=OFF
|
||||
- --max-connections=60
|
||||
- --innodb-buffer-pool-size=512M
|
||||
- --innodb-redo-log-capacity=128M
|
||||
volumes: [mysql_data:/var/lib/mysql]
|
||||
networks: [database]
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, "MYSQL_PWD=$$(cat /run/secrets/mysql_root_password) mysql --user=root --batch --skip-column-names -e 'SELECT 1' >/dev/null"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
start_period: 60s
|
||||
stop_grace_period: 2m
|
||||
mem_limit: 2g
|
||||
logging: *bounded-logging
|
||||
|
||||
app:
|
||||
image: quality-center-app:${QUALITY_IMAGE_TAG:-local}
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: deploy/docker/Dockerfile.app
|
||||
platform: linux/amd64
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
mysql: {condition: service_healthy}
|
||||
environment:
|
||||
MYSQL_DATABASE: ${MYSQL_DATABASE:-quality_platform}
|
||||
MYSQL_USER: ${MYSQL_USER:-quality_app}
|
||||
QUALITY_ADMIN_USERNAME: ${QUALITY_ADMIN_USERNAME:-admin}
|
||||
MYSQL_LEGACY_TIMESTAMP_OFFSET_SECONDS: ${MYSQL_LEGACY_TIMESTAMP_OFFSET_SECONDS:-UNCONFIRMED}
|
||||
QUALITY_ANALYSIS_PROTOCOL_V1_ENABLED: ${QUALITY_ANALYSIS_PROTOCOL_V1_ENABLED:-false}
|
||||
QUALITY_ANALYSIS_AUTONOMOUS_ENABLED: ${QUALITY_ANALYSIS_AUTONOMOUS_ENABLED:-false}
|
||||
JAVA_MAX_HEAP: ${JAVA_MAX_HEAP:-2048m}
|
||||
secrets: [mysql_root_password, mysql_password, admin_password]
|
||||
volumes:
|
||||
- state:/srv/quality/.runtime
|
||||
- archives:/srv/quality/runtimedata/archive
|
||||
- backups:/srv/quality/output/backups
|
||||
- logs:/srv/quality/output/logs
|
||||
networks: [database, frontend]
|
||||
read_only: true
|
||||
tmpfs: [/tmp:mode=1777,size=536870912]
|
||||
cap_drop: [ALL]
|
||||
security_opt: [no-new-privileges:true]
|
||||
healthcheck:
|
||||
test: [CMD, /opt/nao/bin/python, deploy/docker/app.py, health]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 6
|
||||
start_period: 15m
|
||||
stop_grace_period: 5m
|
||||
mem_limit: 6g
|
||||
logging: *bounded-logging
|
||||
|
||||
web:
|
||||
image: quality-center-web:${QUALITY_IMAGE_TAG:-local}
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: deploy/docker/Dockerfile.web
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
app: {condition: service_healthy}
|
||||
ports: ["${HTTP_BIND:-127.0.0.1}:${HTTP_PORT:-5280}:8080"]
|
||||
networks: [frontend]
|
||||
healthcheck:
|
||||
test: [CMD, wget, -q, -O, /dev/null, http://127.0.0.1:8080/healthz]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
mem_limit: 256m
|
||||
logging: *bounded-logging
|
||||
|
||||
networks:
|
||||
database: {internal: true}
|
||||
frontend: {}
|
||||
volumes:
|
||||
mysql_data: {}
|
||||
state: {}
|
||||
archives: {}
|
||||
backups: {}
|
||||
logs: {}
|
||||
secrets:
|
||||
mysql_root_password: {environment: QUALITY_SECRET_MYSQL_ROOT}
|
||||
mysql_password: {environment: QUALITY_SECRET_MYSQL_APP}
|
||||
admin_password: {environment: QUALITY_SECRET_ADMIN}
|
||||
@@ -0,0 +1,27 @@
|
||||
"""构建时下载并校验固定 Linux x86_64 运行时,运行阶段不再下载。"""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import tarfile
|
||||
import tempfile
|
||||
import urllib.request
|
||||
|
||||
|
||||
def install(lock, destination):
|
||||
destination = Path(destination)
|
||||
for asset in json.loads(Path(lock).read_text()):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
temporary = Path(temporary); archive = temporary / 'runtime.tar'
|
||||
with urllib.request.urlopen(asset['url'], timeout=120) as source, archive.open('wb') as output:
|
||||
shutil.copyfileobj(source, output)
|
||||
with archive.open('rb') as source: digest = hashlib.file_digest(source, 'sha256').hexdigest()
|
||||
if digest != asset['sha256']: raise ValueError('Runtime checksum mismatch: ' + asset['name'])
|
||||
unpacked = temporary / 'unpacked'; unpacked.mkdir()
|
||||
with tarfile.open(archive) as source: source.extractall(unpacked, filter='data')
|
||||
children = list(unpacked.iterdir())
|
||||
if len(children) != 1 or not children[0].is_dir(): raise ValueError('Unexpected archive layout')
|
||||
shutil.copytree(children[0], destination / asset['name'], symlinks=True)
|
||||
|
||||
|
||||
if __name__ == '__main__': install('/build/assets.lock.json', '/runtimes')
|
||||
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env python3
|
||||
"""统一 Docker Compose 入口;密码只从本机私有文件加载,不进入源码或命令参数。"""
|
||||
import argparse
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import secrets
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
SECRET_NAMES = {'mysql_root_password': 'QUALITY_SECRET_MYSQL_ROOT',
|
||||
'mysql_password': 'QUALITY_SECRET_MYSQL_APP', 'admin_password': 'QUALITY_SECRET_ADMIN'}
|
||||
|
||||
|
||||
def initialize(root=ROOT):
|
||||
base = Path(root) / '.runtime/docker'
|
||||
directory = base / 'secrets'
|
||||
if base.is_symlink() or directory.is_symlink(): raise ValueError('Unsafe private settings path')
|
||||
directory.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
base.chmod(0o700); directory.chmod(0o700)
|
||||
for name in SECRET_NAMES:
|
||||
path = directory / name
|
||||
if path.is_symlink(): raise ValueError('Unsafe secret file')
|
||||
if path.exists(): continue
|
||||
fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
||||
with os.fdopen(fd, 'w') as stream: stream.write(secrets.token_urlsafe(32) + '\n')
|
||||
config = base / 'config.env'
|
||||
if not config.exists():
|
||||
shutil.copyfile(Path(root) / 'deploy/docker/.env.example', config); config.chmod(0o600)
|
||||
return base
|
||||
|
||||
|
||||
def environment(root=ROOT):
|
||||
env = os.environ.copy(); directory = Path(root) / '.runtime/docker/secrets'
|
||||
for name, variable in SECRET_NAMES.items():
|
||||
path = directory / name
|
||||
if path.is_symlink() or not path.is_file() or path.stat().st_mode & 0o077:
|
||||
raise ValueError('Missing or unsafe secret file; run init first')
|
||||
value = path.read_text().strip()
|
||||
if not re.fullmatch(r'[A-Za-z0-9_-]{24,128}', value): raise ValueError('Invalid secret file')
|
||||
env[variable] = value
|
||||
return env
|
||||
|
||||
|
||||
def command(action):
|
||||
actions = {'config': ['config', '--quiet'], 'build': ['build', '--pull'], 'up': ['up', '-d', '--wait', '--wait-timeout', '1200'],
|
||||
'stop': ['stop'], 'status': ['ps'], 'logs': ['logs', '--tail', '100'],
|
||||
'backup': ['exec', '-T', 'app', '/opt/nao/bin/python', 'scripts/backup.py', 'create-and-verify'],
|
||||
'review-retention': ['exec', '-T', 'app', '/opt/nao/bin/python', 'deploy/docker/app.py', 'review-retention']}
|
||||
return ['docker', 'compose', '-f', str(ROOT / 'deploy/docker/compose.yaml'),
|
||||
'--env-file', str(ROOT / '.runtime/docker/config.env'), *actions[action]]
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action', choices=['init', 'config', 'build', 'up', 'stop', 'status', 'logs', 'backup', 'review-retention'])
|
||||
args = parser.parse_args()
|
||||
if args.action == 'init':
|
||||
base = initialize(); print('Private settings prepared: ' + str(base)); return
|
||||
if not shutil.which('docker'): parser.error('Docker Engine and Compose V2 are required')
|
||||
subprocess.run(command(args.action), cwd=ROOT, env=environment(), check=True)
|
||||
|
||||
|
||||
if __name__ == '__main__': main()
|
||||
@@ -0,0 +1,21 @@
|
||||
server {
|
||||
listen 8080;
|
||||
server_name _;
|
||||
root /usr/share/nginx/html;
|
||||
client_max_body_size 260m;
|
||||
resolver 127.0.0.11 valid=10s;
|
||||
set $quality_backend http://app:5281;
|
||||
location = /healthz { access_log off; proxy_pass $quality_backend/actuator/health; }
|
||||
location /api/ {
|
||||
proxy_pass $quality_backend$request_uri;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header Connection '';
|
||||
proxy_buffering off;
|
||||
proxy_read_timeout 1800s;
|
||||
proxy_send_timeout 1800s;
|
||||
}
|
||||
location / { try_files $uri $uri/ /index.html; }
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,148 @@
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path[:0] = [str(ROOT / 'scripts'), str(ROOT / 'deploy/macos')]
|
||||
|
||||
|
||||
def module(name, path):
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
result = importlib.util.module_from_spec(spec); spec.loader.exec_module(result)
|
||||
return result
|
||||
|
||||
|
||||
app = module('container_app', Path(__file__).with_name('app.py'))
|
||||
manage = module('container_manage', Path(__file__).with_name('manage.py'))
|
||||
import backup
|
||||
import backup_retention as br
|
||||
|
||||
|
||||
class ContainerOperationsTests(unittest.TestCase):
|
||||
def test_retirement_rename_stays_on_the_backup_volume_and_resumes_cleanup(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root=Path(directory).resolve();folder=root/'output/backups/20261110T010000Z-12345678'
|
||||
folder.mkdir(parents=True);(folder/'snapshot').write_bytes(b'old backup')
|
||||
keep=folder.parent/'unrelated';keep.mkdir()
|
||||
br.retire_folder(root,folder,'backup')
|
||||
self.assertFalse(folder.exists())
|
||||
record=next((root/'.runtime/retention/retired').glob('*.json'))
|
||||
value=json.loads(record.read_text());self.assertEqual(value['storage'],'backup-volume')
|
||||
self.assertTrue((root/'output/backups/.retired'/value['target']).is_dir())
|
||||
self.assertEqual(br.clean_retired(root),1);self.assertTrue(keep.is_dir())
|
||||
self.assertFalse(record.exists())
|
||||
|
||||
def test_secret_initialization_is_private_idempotent_and_not_in_command_arguments(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory); (root / 'deploy/docker').mkdir(parents=True)
|
||||
(root / 'deploy/docker/.env.example').write_text('HTTP_BIND=127.0.0.1\n')
|
||||
base = manage.initialize(root)
|
||||
before = {p.name: p.read_bytes() for p in (base / 'secrets').iterdir()}
|
||||
manage.initialize(root)
|
||||
self.assertEqual(before, {p.name: p.read_bytes() for p in (base / 'secrets').iterdir()})
|
||||
for path in (base / 'secrets').iterdir(): self.assertEqual(path.stat().st_mode & 0o077, 0)
|
||||
env = manage.environment(root)
|
||||
self.assertEqual(len(set(env[k] for k in manage.SECRET_NAMES.values())), 3)
|
||||
args = manage.command('up')
|
||||
for value in before.values(): self.assertNotIn(value.decode().strip(), str(args))
|
||||
self.assertNotIn('-v', manage.command('stop'))
|
||||
|
||||
def test_config_uses_tcp_and_refuses_uncoordinated_secret_changes(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
with patch.object(app, 'secret', return_value='a' * 32), patch.dict(os.environ, {}, clear=True):
|
||||
first = app.prepare(root)
|
||||
self.assertEqual(first['mysqlHost'], 'mysql'); self.assertFalse(first['binaryLogEnabled'])
|
||||
self.assertEqual((root / '.runtime/local-config.json').stat().st_mode & 0o077, 0)
|
||||
env = app.backend_environment(first)
|
||||
self.assertIn('jdbc:mysql://mysql:3306/', env['MYSQL_URL'])
|
||||
self.assertEqual(env['SERVER_ADDRESS'], '0.0.0.0')
|
||||
with patch.object(app, 'secret', return_value='b' * 32):
|
||||
with self.assertRaises(ValueError): app.prepare(root)
|
||||
|
||||
def test_backend_exit_causes_container_failure_instead_of_leaving_a_dead_api(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory); (root / '.runtime').mkdir()
|
||||
child = Mock(pid=777); child.poll.return_value = 2
|
||||
with patch.object(app, 'ROOT', root), patch.object(app, 'prepare', return_value={}), \
|
||||
patch.object(app, 'backend_environment', return_value={}), patch.object(br, 'accepted_backend'), \
|
||||
patch.object(app.subprocess, 'Popen', return_value=child), patch.object(app.signal, 'signal'):
|
||||
self.assertEqual(app.serve(), 1)
|
||||
child.wait.assert_called_once()
|
||||
|
||||
def test_shutdown_signals_backend_and_workers_before_waiting(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory); (root / '.runtime').mkdir()
|
||||
children = [Mock(pid=100 + i) for i in range(3)]
|
||||
for child in children: child.poll.return_value = None
|
||||
handlers = {}
|
||||
def register(signum, handler): handlers[signum] = handler
|
||||
def tick(_): handlers[app.signal.SIGTERM](app.signal.SIGTERM, None)
|
||||
with patch.object(app, 'ROOT', root), patch.object(app, 'prepare', return_value={}), \
|
||||
patch.object(app, 'backend_environment', return_value={}), patch.object(br, 'accepted_backend'), \
|
||||
patch.object(app.subprocess, 'Popen', side_effect=children), patch.object(app.signal, 'signal', side_effect=register), \
|
||||
patch.object(app, 'ready', return_value=True), patch.object(app, 'review_retention'), \
|
||||
patch.object(app.time, 'sleep', side_effect=tick), patch.object(app.os, 'killpg') as kill:
|
||||
self.assertEqual(app.serve(), 0)
|
||||
self.assertEqual(kill.call_count, 3)
|
||||
for child in children: child.wait.assert_called_once()
|
||||
|
||||
def test_container_pinned_jar_rejects_corruption(self):
|
||||
with tempfile.TemporaryDirectory() as directory, patch.dict(os.environ, {'QUALITY_CONTAINER': '1'}):
|
||||
root = Path(directory); (root / 'app').mkdir()
|
||||
jar = root / 'app/backend.jar'; jar.write_bytes(b'accepted')
|
||||
(root / 'app/backend.sha256').write_text(br.sha(jar))
|
||||
self.assertEqual(br.accepted_backend(root)[0], jar.resolve())
|
||||
jar.write_bytes(b'changed')
|
||||
with self.assertRaises(ValueError): br.accepted_backend(root)
|
||||
|
||||
def test_backup_uses_owned_private_instance_for_container_and_owns_temporary_files(self):
|
||||
from contextlib import contextmanager
|
||||
import backup_verifier
|
||||
configuration = Path('/tmp/owned-verifier/client.cnf')
|
||||
@contextmanager
|
||||
def instance(root): yield configuration
|
||||
seen = []
|
||||
def verify(folder):
|
||||
seen.append((backup.client_config(), backup.verification_temp_root()))
|
||||
return 'verified'
|
||||
with tempfile.TemporaryDirectory() as directory, patch.object(backup, 'ROOT', Path(directory)), \
|
||||
patch.dict(os.environ, {'QUALITY_CONTAINER': '1'}), patch.object(backup_verifier, 'instance', instance), \
|
||||
patch.object(backup, '_verify', side_effect=verify):
|
||||
self.assertEqual(backup.verify('unused'), 'verified')
|
||||
self.assertEqual(seen, [(configuration, configuration.parent)])
|
||||
self.assertIsNone(backup.CLIENT_CONFIG_OVERRIDE.get())
|
||||
|
||||
def test_compose_preserves_volumes_bounded_logs_secrets_and_private_networks(self):
|
||||
try: import yaml
|
||||
except ImportError: self.skipTest('Install PyYAML for Compose contract checks')
|
||||
compose = yaml.safe_load((ROOT / 'deploy/docker/compose.yaml').read_text())
|
||||
services = compose['services']
|
||||
self.assertNotIn('ports', services['mysql']); self.assertNotIn('ports', services['app'])
|
||||
self.assertEqual(services['mysql']['command'][0], '--skip-log-bin')
|
||||
self.assertEqual(len(compose['volumes']), 5)
|
||||
self.assertEqual(services['app']['depends_on']['mysql']['condition'], 'service_healthy')
|
||||
for service in services.values():
|
||||
self.assertEqual(service['restart'], 'unless-stopped')
|
||||
self.assertEqual(service['logging']['options']['max-size'], '10m')
|
||||
self.assertTrue(services['app']['read_only'])
|
||||
self.assertNotIn('docker.sock', json.dumps(compose))
|
||||
self.assertFalse((ROOT / '.gitea/workflows').exists())
|
||||
|
||||
def test_build_context_excludes_live_data_and_private_baseline(self):
|
||||
ignore = (ROOT / '.dockerignore').read_text()
|
||||
self.assertEqual(next(line for line in ignore.splitlines() if line and not line.startswith('#')), '**')
|
||||
for allowed in ('!.runtime', '!runtimedata', '!output', '!质量数据', '!apikey.txt'):
|
||||
self.assertNotIn(allowed, ignore)
|
||||
self.assertIn('**/original-workbook-acceptance.json', ignore)
|
||||
nginx = (ROOT / 'deploy/docker/nginx.conf').read_text()
|
||||
self.assertIn('proxy_buffering off', nginx)
|
||||
self.assertIn('resolver 127.0.0.11', nginx)
|
||||
|
||||
|
||||
if __name__ == '__main__': unittest.main()
|
||||
@@ -0,0 +1,88 @@
|
||||
#!/usr/bin/env python3
|
||||
"""与后端共享 PID 命名空间的低优先级维护任务;不向容器暴露 Docker socket。"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
import shutil
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path[:0] = [str(ROOT / 'scripts'), str(ROOT / 'deploy/macos')]
|
||||
import backup_retention as br
|
||||
import operations
|
||||
import retention
|
||||
|
||||
|
||||
def console_notify(title, message):
|
||||
print(json.dumps({'event': 'capacity-alert', 'title': title, 'message': message}, ensure_ascii=False), flush=True)
|
||||
return {'submitted': True, 'channel': 'container-log'}
|
||||
|
||||
|
||||
def snapshot(root, support=None, now=None):
|
||||
import backup
|
||||
import managed_test_resources as resources
|
||||
import log_retention
|
||||
root = Path(root); now = time.time() if now is None else now
|
||||
cfg = json.loads((root / '.runtime/local-config.json').read_text())
|
||||
rows = backup.client('SELECT TABLE_SCHEMA,COALESCE(SUM(DATA_LENGTH+INDEX_LENGTH),0) FROM information_schema.tables GROUP BY TABLE_SCHEMA;')
|
||||
estimates = {parts[0]: int(parts[1]) for line in rows.splitlines() if len(parts := line.split('\t')) == 2}
|
||||
registered = 0; overdue = 0
|
||||
for path in resources.registry(root).glob('*.json'):
|
||||
item = resources.load(root, path.stem)
|
||||
if not item or not item.get('created'): continue
|
||||
registered += estimates.get(item['database'], 0)
|
||||
if now >= item.get('expiresAt', now + 1): overdue += 1
|
||||
private = sum(operations.allocated(p) for p in (root / '.runtime/retention').glob('verify-*') if p.is_dir() and not p.is_symlink())
|
||||
mysql = estimates.get(cfg['mysqlDatabase'], 0)
|
||||
archive = operations.allocated(root / 'runtimedata/archive'); backups = operations.allocated(root / 'output/backups')
|
||||
native = operations.allocated(root / '.runtime/nao-users')
|
||||
logs = sum(p.stat().st_blocks * 512 for p in log_retention.candidates(root, root / '.runtime/operations'))
|
||||
core = mysql + archive + backups + native + logs
|
||||
free = min(shutil.disk_usage(root / p).free for p in ('.runtime', 'output/backups', 'output/logs', 'runtimedata/archive'))
|
||||
return {'at': now, 'mysqlBytes': mysql, 'mysqlMeasurement': 'information_schema_estimate_excludes_system_redo_undo',
|
||||
'archiveBytes': archive, 'backupBytes': backups, 'logBytes': logs, 'nativeDataBytes': native,
|
||||
'registeredTestBytes': registered, 'privateVerifierBytes': private, 'temporaryBytes': registered + private,
|
||||
'coreBytes': core, 'artifactBytes': 0, 'permanentBytes': core, 'freeBytes': free,
|
||||
'overdueTestResources': overdue}
|
||||
|
||||
|
||||
def once(role):
|
||||
if role == 'monitor':
|
||||
operations.snapshot = snapshot
|
||||
return operations.run(ROOT, support=ROOT / '.runtime/operations', notifier=console_notify)
|
||||
return retention.run(ROOT)
|
||||
|
||||
|
||||
def main():
|
||||
role = sys.argv[1]
|
||||
if role not in ('retention', 'monitor'): raise ValueError('Unknown worker role')
|
||||
try: os.nice(10)
|
||||
except OSError: pass
|
||||
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s')
|
||||
stopping = False
|
||||
def stop(signum, frame):
|
||||
nonlocal stopping
|
||||
stopping = True
|
||||
signal.signal(signal.SIGTERM, stop); signal.signal(signal.SIGINT, stop)
|
||||
while not stopping:
|
||||
try:
|
||||
once(role)
|
||||
except (br.Busy, retention.Busy): pass
|
||||
except Exception as error:
|
||||
if role == 'monitor': operations.failed_run(ROOT, error, notifier=console_notify)
|
||||
else:
|
||||
path = ROOT / '.runtime/retention/state.json'
|
||||
state = json.loads(path.read_text()) if path.exists() else {}
|
||||
state['consecutiveFailures'] = min(1000, state.get('consecutiveFailures', 0) + 1)
|
||||
state['lastError'] = {'time': time.time(), 'type': type(error).__name__}
|
||||
br.atomic(path, state)
|
||||
print(json.dumps({'event': 'maintenance-error', 'role': role, 'errorType': type(error).__name__}), flush=True)
|
||||
for _ in range(300):
|
||||
if stopping: break
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
if __name__ == '__main__': main()
|
||||
@@ -0,0 +1,94 @@
|
||||
# 本机服务自动启动
|
||||
|
||||
本机使用 `com.lyunlab.quality-center-service` LaunchAgent:解锁并登录当前 macOS 用户后启动,持续守护现有 MySQL 和已验收的后端 JAR。SSH 隧道由已有的 `com.lyunlab.quality-center-tunnel` 独立守护。
|
||||
|
||||
启用 FileVault 的电脑在解锁登录前无法运行用户级守护。退出用户会话、电脑关机、休眠或断网期间,不能依靠这台 Mac 持续提供公网服务。需要全天候服务时,按 [Docker 部署说明](../docker/README.md) 将后台和数据库部署到常在线服务器。
|
||||
|
||||
## 管理
|
||||
|
||||
首次安装维护环境时运行 `python3 -m venv .runtime/retention-env`,再使用 `.runtime/retention-env/bin/pip install --require-hashes -r deploy/macos/retention-requirements.txt` 安装固定依赖。该步骤只安装维护工具,不初始化业务数据库。
|
||||
|
||||
在项目目录执行:
|
||||
|
||||
```sh
|
||||
python3 deploy/macos/service.py status
|
||||
python3 deploy/macos/service.py pause
|
||||
python3 deploy/macos/service.py stop
|
||||
python3 deploy/macos/service.py start
|
||||
```
|
||||
|
||||
- `status`:显示守护、进程身份、数据库和接口就绪状态。
|
||||
- `pause`:停用自动启动和守护,保留当前 MySQL / 后端进程,适合维护前使用。
|
||||
- `stop`:停用守护,按后端、MySQL 的顺序正常停服;超时不会强制结束数据库。
|
||||
- `start`:重新启用自动启动和守护,先确认数据库可查询,再启动后端。
|
||||
|
||||
维护期间先暂停守护,再操作服务。直接运行旧的停止脚本后,仍启用的守护会把进程重新拉起。
|
||||
|
||||
## 实现与边界
|
||||
|
||||
- 安装时登记已验收 JAR 的 SHA-256,从 `.runtime/backend-builds/` 启动对应的不可变运行包。
|
||||
- 只使用 `.runtime/local-config.json` 指定的已初始化数据库;缺少数据标记或运行包摘要不符时停止启动,不初始化、不导入数据库。
|
||||
- 每 3 秒检查进程;异常退出后等待 10 秒重启。数据库可查询后才启动后端。端口被未知进程占用或进程身份无法确认时保留现场。
|
||||
- 守护自身由 launchd 的 `KeepAlive` 负责重启。重新启动守护时会核对启动时间、用户、程序和参数,接管仍存活的服务,避免重复启动。
|
||||
- 只对已经退出的进程自动重启。健康检查暂时失败时不强行终止正在导入或查询的数据库。
|
||||
- 守护脚本按摘要复制到用户的 Application Support 目录。项目源文件的后续编辑不会立即改变已安装脚本。
|
||||
- MySQL 的 `binaryLogEnabled` 配置为 false 时,使用 `--skip-log-bin` 启动。事务重做日志和 undo 文件继续保留。历史时间点恢复不可用,恢复依赖最新完整快照。
|
||||
|
||||
启动配置:`~/Library/LaunchAgents/com.lyunlab.quality-center-service.plist`。
|
||||
|
||||
状态和滚动日志:`~/Library/Application Support/QualityCenterService/`。
|
||||
|
||||
进程业务日志:项目的 `output/logs/`。
|
||||
|
||||
重新安装或升级守护脚本时使用 `service.py install --root <项目绝对路径> --jar <已验收JAR绝对路径>`。部署配置中只记录路径与摘要,密码仍从原有私有配置读取。
|
||||
|
||||
## 验证
|
||||
|
||||
```sh
|
||||
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s deploy/macos -p 'test_service.py' -v
|
||||
```
|
||||
|
||||
隔离进程检查覆盖 MySQL 与后端异常退出重启、数据库启动顺序、已有进程接管、未知进程保护和运行包/数据目录前置校验。实际重启电脑和 FileVault 解锁后的启动仍需单独验收。
|
||||
|
||||
## 自动数据保留
|
||||
|
||||
`com.lyunlab.quality-center-retention` 是独立的低优先级维护任务,解锁登录后运行,每5分钟检查一次。它使用当前已验收 JAR,独立运行维护脚本;不重新构建业务后端。
|
||||
|
||||
- **旧版本**:当前正式版本、候选版本及基线、待配置来源的基线、分析会话、已保存报表/快照/清单、巡检等业务引用均保护。其他已发布版本连续30分钟未被引用,且当前版本已有恢复验证通过的备份后,才进入回收。
|
||||
- **旧构建**:只处理已无版本引用的终态构建。分批清理,每批最多1000行;先撤销其可复用状态和唯一构建键,避免新导入复用正在清理的半成品。仍有未知业务引用时保留。
|
||||
- **旧来源修订**:每5分钟检查,已完成修订连续30分钟没有任何版本、构建、明细、来源清单、替换/撤销/恢复记录或保存内容引用后才回收。修订身份变化或重新出现引用会重置保护期;未知状态、缺失或校验异常的文件保留。SHA-256、大小、路径边界与文件身份全部通过后才删除修订记录,原件另行核对共享路径和保存内容引用。共享文件保留;文件删除意图先持久登记,意外退出后可续清,不扫描删除未登记的孤立文件。
|
||||
- **上传草稿**:24小时过期后再留30分钟保护期。已入库文件先核对正式归档摘要,再重绑定来源路径、清理临时副本;待导入来源仍引用的文件保留。无数据库记录的上传临时文件超过24小时才检查回收。
|
||||
- **备份**:检测到新正式版本或距离上次备份超过24小时,在任务空闲时生成新备份。新备份必须通过独立 MySQL 实例恢复、表行数、正式数据汇总、应用文件和来源归档校验,才原子更新最新备份入口并回收旧备份。失败时保留旧有效备份,失败重试间隔至少1小时。
|
||||
- **任务保护**:导入、数据工作流、分析、报告、巡检、排队任务及待自动重试任务存在时,自动维护延期。每一批删除都重新检查任务状态和当前版本;短暂取得数据库排他访问失败时延期,不终止业务查询或任务。
|
||||
- **互斥与失败恢复**:手动备份、恢复和自动维护共享 `.runtime/data-maintenance.lock`。恢复校验使用临时、仅本机私有 socket 可访问的独立 MySQL,完成后正常关闭并删除;异常遗留只根据所有者记录及进程身份处理,不按数据库名前缀批量删除。
|
||||
- **维护暂停**:`service.py pause/stop` 同时写入维护暂停标记;`service.py start/install` 解除。执行中的备份允许安全完成,后续删除批次会停下。
|
||||
- **升级保护**:当前 JAR 摘要或数据库结构发生变化时暂停自动清理,重新检查兼容性后执行维护安装命令。未知任务状态也按进行中处理。
|
||||
- **空间余量**:开始新备份前预留至少三倍数据库表/索引容量、归档空间和额外安全余量;空间不足时保留现有有效备份并延期。失败的新候选只清理本次登记的文件,未知目录不会删除。
|
||||
|
||||
最新备份记录在 `.runtime/retention/latest.json`,实际目录在 `output/backups/`,界面“任务与监控”的备份列表可查看。
|
||||
|
||||
维护任务状态与滚动日志分别在 `.runtime/retention/state.json` 和 `~/Library/Application Support/QualityCenterService/logs/retention.log`。
|
||||
|
||||
来源修订回收计数为 `sourceRevisionsRemoved`、`sourceArchiveFilesRemoved` 和 `sourceArchiveBytesRemoved`,待解除文件引用的登记数为 `sourceFilesPending`。登记目录为 `.runtime/retention/source-files/`。来源清单中仍可重新导入的记录(包括已导入状态)属于有效引用,不能作为无引用修订删除。
|
||||
|
||||
```sh
|
||||
.runtime/retention-env/bin/python3 deploy/macos/retention.py status --root "$PWD"
|
||||
.runtime/retention-env/bin/python3 deploy/macos/retention.py install --root "$PWD"
|
||||
PYTHONDONTWRITEBYTECODE=1 .runtime/retention-env/bin/python3 -m unittest discover -s deploy/macos -p 'test_retention.py' -v
|
||||
```
|
||||
|
||||
维护依赖 PyMySQL 固定版本及 SHA-256 在 `retention-requirements.txt` 中。私有 Python 环境与依赖不修改全局安装。日常删除释放的 InnoDB 表内空间由后续导入复用;必要的物理表空间整理另在低峰进行。
|
||||
|
||||
## 容量、测试资源与日志
|
||||
|
||||
`operations.py` 安装独立的 5 分钟检查任务。容量阈值为 30/50GiB,余量阈值为 50/20GiB;监测业务数据、交付产物、临时校验空间及约 24 小时增长,保留 30 天日报。告警通过 macOS 系统通知发送,并记录在 `.runtime/operations/`,相同告警 6 小时内去重。
|
||||
|
||||
注册过的测试数据库在正常退出时回收;异常遗留超过 24 小时,确认所有者和子进程退出、标记一致且无连接后才删除。未登记数据库继续保留。日志按 7 天、每类 100MiB 限制;Java 原生日志滚动配置在 `logback-quality.xml`,MySQL 通过重新打开错误日志完成滚动,不重启数据库,不截断活动文件。
|
||||
|
||||
```sh
|
||||
.runtime/retention-env/bin/python3 deploy/macos/operations.py install --root "$PWD"
|
||||
.runtime/retention-env/bin/python3 deploy/macos/operations.py status --root "$PWD"
|
||||
.runtime/retention-env/bin/python3 deploy/macos/operations.py notify-test --root "$PWD"
|
||||
```
|
||||
|
||||
安装日志配置后,后端需在任务空闲时正常重启一次才能加载;不要为加载日志配置中断导入。Linux 复用同一回收逻辑,采用容器日志告警,见 Docker 说明。
|
||||
@@ -0,0 +1,19 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<configuration>
|
||||
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
|
||||
<property name="QUALITY_LOG_DIR" value="${QUALITY_LOG_DIR:-output/logs}"/>
|
||||
<appender name="QUALITY_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
|
||||
<file>${QUALITY_LOG_DIR}/backend.log</file>
|
||||
<append>true</append>
|
||||
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
|
||||
<fileNamePattern>${QUALITY_LOG_DIR}/backend.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
|
||||
<maxFileSize>10MB</maxFileSize>
|
||||
<maxHistory>7</maxHistory>
|
||||
<totalSizeCap>80MB</totalSizeCap>
|
||||
<cleanHistoryOnStart>true</cleanHistoryOnStart>
|
||||
</rollingPolicy>
|
||||
<encoder><pattern>${FILE_LOG_PATTERN}</pattern><charset>UTF-8</charset></encoder>
|
||||
</appender>
|
||||
<logger name="org.springframework.security" level="WARN"/>
|
||||
<root level="INFO"><appender-ref ref="QUALITY_FILE"/></root>
|
||||
</configuration>
|
||||
@@ -0,0 +1,276 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Independent capacity monitoring, registered-test cleanup and log retention."""
|
||||
import argparse
|
||||
import fcntl
|
||||
from datetime import datetime
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
from logging.handlers import RotatingFileHandler
|
||||
import os
|
||||
from pathlib import Path
|
||||
import plistlib
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
LABEL = 'com.lyunlab.quality-center-operations'
|
||||
SUPPORT = Path.home() / 'Library/Application Support/QualityCenterService'
|
||||
GIB = 1024**3
|
||||
DEFAULTS = {'intervalSeconds': 300, 'warningBytes': 30 * GIB, 'criticalBytes': 50 * GIB,
|
||||
'freeWarningBytes': 50 * GIB, 'freeCriticalBytes': 20 * GIB,
|
||||
'growthBytes': 2 * GIB, 'growthFraction': .2, 'notificationCooldownSeconds': 21600,
|
||||
'logMaxAgeSeconds': 604800, 'logClassMaxBytes': 100 * 1024**2,
|
||||
'testMaxAgeSeconds': 86400}
|
||||
|
||||
|
||||
def allocated(path):
|
||||
path = Path(path)
|
||||
if not path.exists() or path.is_symlink(): return 0
|
||||
result = subprocess.run(['/usr/bin/du', '-sk', str(path)], capture_output=True, text=True, timeout=45)
|
||||
lines = result.stdout.splitlines()
|
||||
if result.returncode or not lines: raise RuntimeError('Capacity scan incomplete')
|
||||
return int(lines[-1].split()[0]) * 1024
|
||||
|
||||
|
||||
def snapshot(root, support=SUPPORT, now=None):
|
||||
import managed_test_resources as resources
|
||||
import log_retention
|
||||
root = Path(root).resolve(); now = time.time() if now is None else now
|
||||
mysql = allocated(root / '.runtime/mysql-data')
|
||||
archives = allocated(root / 'runtimedata/archive'); backups = allocated(root / 'output/backups')
|
||||
runtime = allocated(root / '.runtime'); outputs = allocated(root / 'output')
|
||||
logs = log_retention.candidates(root, support)
|
||||
log_bytes = sum(p.stat().st_blocks * 512 for p in logs)
|
||||
output_logs = sum(p.stat().st_blocks * 512 for p in logs if p.is_relative_to(root / 'output'))
|
||||
runtime_logs = sum(p.stat().st_blocks * 512 for p in logs if p.is_relative_to(root / '.runtime'))
|
||||
native = allocated(root / '.runtime/nao-users')
|
||||
registered = 0; overdue = 0
|
||||
for p in resources.registry(root).glob('*.json'):
|
||||
value = resources.load(root, p.stem)
|
||||
if not value or not value.get('created'): continue
|
||||
if now >= value.get('expiresAt', now + 1): overdue += 1
|
||||
registered += allocated(root / '.runtime/mysql-data' / value['database'])
|
||||
private = 0
|
||||
for p in (root / '.runtime/retention').glob('verify-*'):
|
||||
if p.is_symlink(): continue
|
||||
try: owner = json.loads((p / 'owner.json').read_text())
|
||||
except (OSError, ValueError): continue
|
||||
if owner.get('kind') == 'private-quality-restore' and owner.get('directory') == str(p): private += allocated(p)
|
||||
# Core service data is the same scope as the earlier 11.86GiB measurement.
|
||||
# Software/build/delivery artifacts are visible separately and included in growth checks.
|
||||
core = max(0, mysql - registered) + archives + backups + log_bytes + native
|
||||
artifacts = max(0, runtime - mysql - private - native - runtime_logs) + max(0, outputs - backups - output_logs)
|
||||
return {'at': now, 'mysqlBytes': mysql, 'archiveBytes': archives, 'backupBytes': backups,
|
||||
'logBytes': log_bytes, 'nativeDataBytes': native, 'registeredTestBytes': registered,
|
||||
'privateVerifierBytes': private, 'temporaryBytes': registered + private,
|
||||
'coreBytes': core, 'artifactBytes': artifacts, 'permanentBytes': core + artifacts,
|
||||
'freeBytes': shutil.disk_usage(root).free, 'overdueTestResources': overdue}
|
||||
|
||||
|
||||
def conditions(sample, policy, maintenance, history):
|
||||
alerts = {}
|
||||
if sample['coreBytes'] >= policy['criticalBytes']:
|
||||
alerts['capacity'] = {'severity': 'critical', 'message': f"业务数据占用已达{sample['coreBytes']/GIB:.1f}GiB,请检查增长来源。"}
|
||||
elif sample['coreBytes'] >= policy['warningBytes']:
|
||||
alerts['capacity'] = {'severity': 'warning', 'message': f"业务数据占用已达{sample['coreBytes']/GIB:.1f}GiB。"}
|
||||
for key, label, size in [('temporary', '测试及校验临时空间', sample.get('temporaryBytes', 0)),
|
||||
('artifacts', '交付及构建产物', sample.get('artifactBytes', max(0, sample['permanentBytes'] - sample['coreBytes'])))]:
|
||||
if size >= policy['warningBytes']:
|
||||
alerts[key] = {'severity': 'critical' if size >= policy['criticalBytes'] else 'warning',
|
||||
'message': f'{label}已占用{size/GIB:.1f}GiB,请核对;正在运行的任务会继续保护。'}
|
||||
if sample['freeBytes'] < policy['freeCriticalBytes']:
|
||||
alerts['disk'] = {'severity': 'critical', 'message': f"磁盘仅余{sample['freeBytes']/GIB:.1f}GiB。"}
|
||||
elif sample['freeBytes'] < policy['freeWarningBytes']:
|
||||
alerts['disk'] = {'severity': 'warning', 'message': f"磁盘剩余{sample['freeBytes']/GIB:.1f}GiB,请预留备份空间。"}
|
||||
if maintenance.get('consecutiveFailures', 0) >= 3:
|
||||
alerts['maintenance'] = {'severity': 'critical', 'message': '数据维护已连续失败3次,请检查维护记录。'}
|
||||
if sample['overdueTestResources']:
|
||||
alerts['tests'] = {'severity': 'warning', 'message': f"有{sample['overdueTestResources']}个测试资源已超过24小时;正在运行或身份未确认的资源将继续保留。"}
|
||||
old = [v for v in history if 21 * 3600 <= sample['at'] - v['at'] <= 27 * 3600]
|
||||
if old:
|
||||
baseline = min(old, key=lambda x: abs(sample['at'] - x['at'] - 86400))
|
||||
changes = []
|
||||
for label, key in [('业务数据', 'coreBytes'), ('交付及构建产物', 'artifactBytes')]:
|
||||
before = baseline.get(key, baseline['permanentBytes'] - baseline['coreBytes'])
|
||||
after = sample.get(key, sample['permanentBytes'] - sample['coreBytes'])
|
||||
delta = after - before
|
||||
if delta > max(policy['growthBytes'], before * policy['growthFraction']):
|
||||
changes.append(f'{label}增长{delta/GIB:.1f}GiB')
|
||||
if changes:
|
||||
alerts['growth'] = {'severity': 'warning', 'message': '约24小时' + ','.join(changes) + ',请核对。'}
|
||||
return alerts
|
||||
|
||||
|
||||
def notify(title, message):
|
||||
script = 'on run argv\ndisplay notification (item 2 of argv) with title (item 1 of argv)\nend run\n'
|
||||
try:
|
||||
p = subprocess.run(['/usr/bin/osascript', '-', title, message], input=script, text=True,
|
||||
capture_output=True, timeout=10)
|
||||
return {'submitted': p.returncode == 0, 'channel': 'macOS', 'exitCode': p.returncode}
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return {'submitted': False, 'channel': 'macOS'}
|
||||
|
||||
|
||||
def dispatch(alerts, previous, now, policy, notifier=notify):
|
||||
updated = {}; events = []
|
||||
for key, value in alerts.items():
|
||||
prior = previous.get(key, {})
|
||||
due = (not prior or not prior.get('lastSubmittedAt') or prior.get('severity') != value['severity'] or
|
||||
now - prior.get('lastSubmittedAt', 0) >= policy['notificationCooldownSeconds'])
|
||||
entry = {**value, 'since': prior.get('since', now), 'lastSubmittedAt': prior.get('lastSubmittedAt', 0)}
|
||||
if due:
|
||||
result = notifier('质量平台容量告警', value['message'])
|
||||
entry['notification'] = result
|
||||
# Failed submission retries on the next check; duplicates are limited after success.
|
||||
if result.get('submitted'): entry['lastSubmittedAt'] = now
|
||||
events.append({'kind': 'alert', 'key': key, **value, 'at': now, 'notification': result})
|
||||
updated[key] = entry
|
||||
for key in set(previous) - set(alerts):
|
||||
events.append({'kind': 'resolved', 'key': key, 'at': now})
|
||||
return updated, events
|
||||
|
||||
|
||||
def janitor(root, support, policy):
|
||||
import backup_retention as br
|
||||
import backup_verifier
|
||||
import managed_test_resources as resources
|
||||
import log_retention
|
||||
from retention import Database, Busy
|
||||
root = Path(root).resolve()
|
||||
if (root / '.runtime/retention/paused').exists(): return {'deferred': 'Maintenance is paused'}
|
||||
try:
|
||||
with br.lock(root, timeout=0):
|
||||
db = Database(root)
|
||||
try:
|
||||
config = json.loads((root / '.runtime/retention/config.json').read_text())
|
||||
accepted = br.accepted_backend(root)
|
||||
if config['schemaSignature'] != db.schema_signature or not accepted or config['acceptedJarSha256'] != accepted[1]:
|
||||
return {'deferred': 'Application version changed'}
|
||||
db.current()
|
||||
if db.busy(): return {'deferred': 'Application jobs are active'}
|
||||
# Use a short-lived CLI connection so no observer holds a test schema open.
|
||||
import backup
|
||||
result = resources.cleanup(root, backup.client)
|
||||
result['abandonedVerifiersRemoved'] = backup_verifier.clean_abandoned(root)
|
||||
result['mysqlLogsRotated'] = log_retention.rotate_mysql(root, backup.client, br)
|
||||
result.update(log_retention.retain(root, support,
|
||||
max_bytes=policy['logClassMaxBytes'], max_age=policy['logMaxAgeSeconds']))
|
||||
return result
|
||||
finally: db.conn.close()
|
||||
except (br.Busy, Busy): return {'deferred': 'Another maintenance or application operation is active'}
|
||||
|
||||
|
||||
def run(root, support=SUPPORT, notifier=notify):
|
||||
import backup_retention as br
|
||||
root = Path(root).resolve(); base = root / '.runtime/operations'
|
||||
if base.is_symlink() or (root / '.runtime').is_symlink(): raise ValueError('Unsafe operations state path')
|
||||
base.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
policy = {**DEFAULTS, **json.loads((base / 'policy.json').read_text())}
|
||||
state_path = base / 'monitor.json'; previous = json.loads(state_path.read_text()) if state_path.exists() else {}
|
||||
history_path = base / 'samples.json'; history = json.loads(history_path.read_text()) if history_path.exists() else []
|
||||
now = time.time(); sample = snapshot(root, support, now)
|
||||
maintenance_path = root / '.runtime/retention/state.json'
|
||||
maintenance = json.loads(maintenance_path.read_text()) if maintenance_path.exists() else {}
|
||||
alerts = conditions(sample, policy, maintenance, history)
|
||||
cleanup = {}; failures = previous.get('cleanupFailures', 0)
|
||||
try:
|
||||
cleanup = janitor(root, support, policy)
|
||||
if 'deferred' not in cleanup: failures = 0
|
||||
except Exception as error:
|
||||
failures += 1; cleanup = {'errorType': type(error).__name__}
|
||||
if failures >= 3: alerts['cleanup'] = {'severity': 'warning', 'message': '日志或测试环境清理已连续失败3次,请检查维护记录。'}
|
||||
active, events = dispatch(alerts, previous.get('alerts', {}), now, policy, notifier)
|
||||
state = {'checkedAt': now, 'sample': sample, 'cleanup': cleanup, 'cleanupFailures': failures, 'alerts': active}
|
||||
br.atomic(state_path, state)
|
||||
(base / 'failure.json').unlink(missing_ok=True)
|
||||
history = [v for v in history if now - v['at'] <= 27 * 3600]
|
||||
history.append(sample); br.atomic(history_path, history[-330:])
|
||||
daily_path = base / 'daily.json'; daily = json.loads(daily_path.read_text()) if daily_path.exists() else {}
|
||||
day = datetime.fromtimestamp(now, ZoneInfo('Asia/Shanghai')).date().isoformat(); daily[day] = sample
|
||||
br.atomic(daily_path, {k: daily[k] for k in sorted(daily)[-30:]})
|
||||
if events:
|
||||
handler = RotatingFileHandler(base / 'alerts.jsonl', maxBytes=1024**2, backupCount=2)
|
||||
handler.setFormatter(logging.Formatter('%(message)s'))
|
||||
logger = logging.getLogger('quality-operations-alerts'); logger.addHandler(handler); logger.setLevel(logging.INFO); logger.propagate = False
|
||||
try:
|
||||
for event in events: logger.info(json.dumps(event, ensure_ascii=False))
|
||||
finally: logger.removeHandler(handler); handler.close()
|
||||
return state
|
||||
|
||||
|
||||
def failed_run(root, error, notifier=notify):
|
||||
import backup_retention as br
|
||||
base = Path(root).resolve() / '.runtime/operations'
|
||||
if base.is_symlink() or base.parent.is_symlink(): return
|
||||
path = base / 'failure.json'; value = json.loads(path.read_text()) if path.exists() else {}
|
||||
now = time.time(); value.update({'count': value.get('count', 0) + 1, 'time': now, 'type': type(error).__name__})
|
||||
if value['count'] >= 3 and now - value.get('lastSubmittedAt', 0) >= DEFAULTS['notificationCooldownSeconds']:
|
||||
result = notifier('质量平台监测异常', '容量监测已连续失败3次,请查看监测状态。')
|
||||
value['notification'] = result
|
||||
if result.get('submitted'): value['lastSubmittedAt'] = now
|
||||
br.atomic(path, value)
|
||||
|
||||
|
||||
def install(root):
|
||||
import backup_retention as br
|
||||
root = Path(root).resolve(); base = root / '.runtime/operations'
|
||||
base.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
policy_path = base / 'policy.json'
|
||||
policy = {**DEFAULTS, **(json.loads(policy_path.read_text()) if policy_path.exists() else {})}
|
||||
br.atomic(policy_path, policy)
|
||||
files = [Path(__file__).resolve(), root / 'deploy/macos/retention.py'] + [root / 'scripts' / p for p in
|
||||
('backup_retention.py', 'revision_retention.py', 'backup_verifier.py', 'managed_test_resources.py', 'log_retention.py')]
|
||||
digest = hashlib.sha256(b''.join(p.read_bytes() for p in files)).hexdigest()
|
||||
version = SUPPORT / 'operations' / digest; version.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
for p in files: shutil.copy2(p, version / p.name); (version / p.name).chmod(0o600)
|
||||
log_config = root / 'deploy/macos/logback-quality.xml'
|
||||
shutil.copy2(log_config, base / 'logback-quality.xml'); (base / 'logback-quality.xml').chmod(0o600)
|
||||
logs = SUPPORT / 'logs'; logs.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
value = {'Label': LABEL, 'ProgramArguments': [str(root / '.runtime/retention-env/bin/python3'), str(version / 'operations.py'), 'run', '--root', str(root)],
|
||||
'RunAtLoad': True, 'StartInterval': 300, 'WorkingDirectory': str(root), 'ProcessType': 'Background',
|
||||
'Nice': 10, 'LowPriorityIO': True, 'EnvironmentVariables': {'PYTHONDONTWRITEBYTECODE': '1'},
|
||||
'StandardOutPath': str(logs / 'operations-launchd.log'), 'StandardErrorPath': str(logs / 'operations-launchd-error.log')}
|
||||
domain = 'gui/' + str(os.getuid()); plist = Path.home() / 'Library/LaunchAgents' / (LABEL + '.plist')
|
||||
subprocess.run(['launchctl', 'bootout', domain + '/' + LABEL], capture_output=True)
|
||||
with plist.open('wb') as f: plistlib.dump(value, f)
|
||||
plist.chmod(0o600); subprocess.run(['launchctl', 'enable', domain + '/' + LABEL], check=True)
|
||||
for _ in range(50):
|
||||
p = subprocess.run(['launchctl', 'bootstrap', domain, str(plist)], capture_output=True)
|
||||
if p.returncode == 0: break
|
||||
time.sleep(.2)
|
||||
else: raise RuntimeError('Operations job registration did not complete')
|
||||
print('Installed independent five-minute capacity and housekeeping checks')
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action', choices=['install', 'run', 'status', 'notify-test']); parser.add_argument('--root', required=True)
|
||||
args = parser.parse_args(); root = Path(args.root).resolve(); sys.path.append(str(root / 'scripts'))
|
||||
import backup_retention as br
|
||||
if args.action == 'install':
|
||||
with br.lock(root, timeout=0): install(root)
|
||||
elif args.action == 'status':
|
||||
print((root / '.runtime/operations/monitor.json').read_text())
|
||||
elif args.action == 'notify-test':
|
||||
print(json.dumps(notify('质量平台通知测试', '容量监测已启用。这是测试通知,当前没有容量告警。'), ensure_ascii=False))
|
||||
else:
|
||||
logs = SUPPORT / 'logs'; logs.mkdir(parents=True, exist_ok=True)
|
||||
handler = RotatingFileHandler(logs / 'operations.log', maxBytes=2 * 1024**2, backupCount=2)
|
||||
handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %(message)s'))
|
||||
logging.getLogger().addHandler(handler); logging.getLogger().setLevel(logging.INFO)
|
||||
with (root / '.runtime/operations/monitor.lock').open('a') as monitor_lock:
|
||||
fcntl.flock(monitor_lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
try:
|
||||
state = run(root)
|
||||
logging.info('operations complete coreGiB=%.3f temporaryGiB=%.3f artifactsGiB=%.3f cleanup=%s alerts=%s',
|
||||
state['sample']['coreBytes'] / GIB, state['sample']['temporaryBytes'] / GIB,
|
||||
state['sample']['artifactBytes'] / GIB, json.dumps(state['cleanup']), list(state['alerts']))
|
||||
except Exception as error:
|
||||
logging.error('operations failed safely: %s', type(error).__name__)
|
||||
failed_run(root, error); raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__': main()
|
||||
@@ -0,0 +1,23 @@
|
||||
PyMySQL==1.2.3 --hash=sha256:14f1c68e2ed859243ae5ca41ffbe677027fc46bc136a9f0be8a4e928e5e7415a
|
||||
psutil==7.2.2 \
|
||||
--hash=sha256:0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372 \
|
||||
--hash=sha256:076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9 \
|
||||
--hash=sha256:11fe5a4f613759764e79c65cf11ebdf26e33d6dd34336f8a337aa2996d71c841 \
|
||||
--hash=sha256:1a571f2330c966c62aeda00dd24620425d4b0cc86881c89861fbc04549e5dc63 \
|
||||
--hash=sha256:1a7b04c10f32cc88ab39cbf606e117fd74721c831c98a27dc04578deb0c16979 \
|
||||
--hash=sha256:1fa4ecf83bcdf6e6c8f4449aff98eefb5d0604bf88cb883d7da3d8d2d909546a \
|
||||
--hash=sha256:2edccc433cbfa046b980b0df0171cd25bcaeb3a68fe9022db0979e7aa74a826b \
|
||||
--hash=sha256:7b6d09433a10592ce39b13d7be5a54fbac1d1228ed29abc880fb23df7cb694c9 \
|
||||
--hash=sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee \
|
||||
--hash=sha256:917e891983ca3c1887b4ef36447b1e0873e70c933afc831c6b6da078ba474312 \
|
||||
--hash=sha256:ab486563df44c17f5173621c7b198955bd6b613fb87c71c161f827d3fb149a9b \
|
||||
--hash=sha256:ae0aefdd8796a7737eccea863f80f81e468a1e4cf14d926bd9b6f5f2d5f90ca9 \
|
||||
--hash=sha256:b0726cecd84f9474419d67252add4ac0cd9811b04d61123054b9fb6f57df6e9e \
|
||||
--hash=sha256:b58fabe35e80b264a4e3bb23e6b96f9e45a3df7fb7eed419ac0e5947c61e47cc \
|
||||
--hash=sha256:c7663d4e37f13e884d13994247449e9f8f574bc4655d509c3b95e9ec9e2b9dc1 \
|
||||
--hash=sha256:e452c464a02e7dc7822a05d25db4cde564444a67e58539a00f929c51eddda0cf \
|
||||
--hash=sha256:e78c8603dcd9a04c7364f1a3e670cea95d51ee865e4efb3556a3a63adef958ea \
|
||||
--hash=sha256:eb7e81434c8d223ec4a219b5fc1c47d0417b12be7ea866e24fb5ad6e84b3d988 \
|
||||
--hash=sha256:ed0cace939114f62738d808fdcecd4c869222507e266e574799e9c0faa17d486 \
|
||||
--hash=sha256:eed63d3b4d62449571547b60578c5b2c4bcccc5387148db46e0c2313dad0ee00 \
|
||||
--hash=sha256:fd04ef36b4a6d599bbdb225dd1d3f51e00105f6d48a28f006da7f9822f2606d8
|
||||
@@ -0,0 +1,525 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Conservative, online retention for this Mac's accepted Quality Center runtime."""
|
||||
import argparse
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime
|
||||
import hashlib
|
||||
import importlib
|
||||
import json
|
||||
import logging
|
||||
from logging.handlers import RotatingFileHandler
|
||||
import os
|
||||
from pathlib import Path
|
||||
import plistlib
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
import pymysql
|
||||
from pymysql.cursors import DictCursor
|
||||
|
||||
LABEL = 'com.lyunlab.quality-center-retention'
|
||||
SUPPORT = Path.home() / 'Library/Application Support/QualityCenterService'
|
||||
TERMINAL = ('COMPLETED', 'FAILED', 'CANCELLED', 'SUCCEEDED', 'STOPPED', 'REJECTED', 'CLOSED', 'INTERRUPTED', 'INVALIDATED')
|
||||
BUILD_CHILDREN = ('defect_component', 'fine_defect_rollup', 'metric_rollup', 'sales_order_metric_rollup',
|
||||
'fine_defect_rollup_build', 'metric_rollup_build', 'sales_order_metric_rollup_build',
|
||||
'fact_black', 'fact_bright')
|
||||
OWNED_RELEASE = ('release_member', 'release_protocol_dependency', 'release_source_exclusion',
|
||||
'release_action', 'candidate_reconciliation')
|
||||
MUTABLE_RELEASE_REFS = ('data_release', 'import_run', 'release_action', 'candidate_reconciliation')
|
||||
|
||||
|
||||
def ident(value):
|
||||
if not re.fullmatch(r'[A-Za-z0-9_]+', value):
|
||||
raise ValueError('Invalid SQL identifier')
|
||||
return '`' + value + '`'
|
||||
|
||||
|
||||
class Busy(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def safe_file(path, boundary):
|
||||
path, boundary = Path(path), Path(boundary).absolute()
|
||||
if not path.is_absolute():
|
||||
return False
|
||||
if not path.is_relative_to(boundary) or not path.resolve().is_relative_to(boundary.resolve()):
|
||||
return False
|
||||
cursor = path
|
||||
while cursor != boundary:
|
||||
if cursor.is_symlink():
|
||||
return False
|
||||
cursor = cursor.parent
|
||||
return not boundary.is_symlink() and path.is_file()
|
||||
|
||||
|
||||
def fingerprint(path):
|
||||
s = Path(path).stat()
|
||||
return (s.st_dev, s.st_ino, s.st_size, s.st_mtime_ns)
|
||||
|
||||
|
||||
class Database:
|
||||
def __init__(self, root, database=None):
|
||||
self.root = Path(root).resolve()
|
||||
self.config = json.loads((self.root / '.runtime/local-config.json').read_text())
|
||||
self.name = database or self.config['mysqlDatabase']
|
||||
ident(self.name)
|
||||
endpoint = ({'host': self.config['mysqlHost'], 'port': int(self.config['mysqlPort'])}
|
||||
if self.config.get('mysqlHost') else {'unix_socket': str(self.root / '.runtime/mysql.sock')})
|
||||
self.conn = pymysql.connect(**endpoint, user='root',
|
||||
password=self.config['rootPassword'], database=self.name, charset='utf8mb4',
|
||||
autocommit=True, cursorclass=DictCursor, connect_timeout=3, read_timeout=10, write_timeout=5)
|
||||
self.execute("SET SESSION time_zone='+00:00'")
|
||||
self.execute('SET SESSION lock_wait_timeout=1')
|
||||
self.execute('SET SESSION innodb_lock_wait_timeout=1')
|
||||
rows = self.rows('SELECT TABLE_NAME,COLUMN_NAME,IS_NULLABLE,DATA_TYPE FROM information_schema.columns WHERE TABLE_SCHEMA=%s', (self.name,))
|
||||
self.columns = {}
|
||||
self.column_types = {}
|
||||
for row in rows:
|
||||
self.columns.setdefault(row['TABLE_NAME'], {})[row['COLUMN_NAME']] = row['IS_NULLABLE'] == 'YES'
|
||||
self.column_types.setdefault(row['TABLE_NAME'], {})[row['COLUMN_NAME']] = row['DATA_TYPE']
|
||||
self.tables = sorted(self.columns)
|
||||
self.fks = self.rows('SELECT TABLE_NAME,COLUMN_NAME,REFERENCED_TABLE_NAME,REFERENCED_COLUMN_NAME FROM information_schema.key_column_usage WHERE TABLE_SCHEMA=%s AND REFERENCED_TABLE_NAME IS NOT NULL', (self.name,))
|
||||
self.schema_signature=hashlib.sha256(json.dumps({'columns':self.columns,'fks':sorted(self.fks,key=lambda r:json.dumps(r,sort_keys=True))},sort_keys=True).encode()).hexdigest()
|
||||
self.longest_exclusion=0.0
|
||||
if not {'current_release','data_release','release_member','transform_build','source_upload_draft'}.issubset(self.columns):
|
||||
raise ValueError('Unexpected application schema')
|
||||
|
||||
def rows(self, sql, args=()):
|
||||
with self.conn.cursor() as cursor:
|
||||
cursor.execute(sql, args)
|
||||
return list(cursor.fetchall())
|
||||
|
||||
def execute(self, sql, args=()):
|
||||
with self.conn.cursor() as cursor:
|
||||
return cursor.execute(sql, args)
|
||||
|
||||
def current(self):
|
||||
rows = self.rows("SELECT release_id FROM current_release WHERE dataset_id='quality'")
|
||||
if len(rows) != 1 or not rows[0]['release_id']:
|
||||
raise Busy('Current release not confirmed')
|
||||
fields='status,complete' if 'complete' in self.columns['data_release'] else 'status'
|
||||
release=self.rows('SELECT '+fields+' FROM data_release WHERE id=%s',(rows[0]['release_id'],))
|
||||
if len(release)!=1 or release[0]['status']!='PUBLISHED' or release[0].get('complete',True) not in (True,1):
|
||||
raise Busy('Current release is not a complete publication')
|
||||
return rows[0]['release_id']
|
||||
|
||||
def busy(self):
|
||||
for table in ('task','import_run','data_workflow_operation','analysis_turn','analysis_protocol_execution','quality_inspection_run'):
|
||||
if table not in self.columns or 'status' not in self.columns[table]:
|
||||
continue
|
||||
sql = 'SELECT 1 FROM ' + ident(table) + ' WHERE (status IS NULL OR status NOT IN (' + ','.join(['%s']*len(TERMINAL)) + '))'
|
||||
if table == 'task' and {'next_retry_at','retry_count','max_retries'}.issubset(self.columns[table]):
|
||||
sql += " OR (status='FAILED' AND next_retry_at IS NOT NULL AND retry_count<max_retries)"
|
||||
if self.rows(sql + ' LIMIT 1', TERMINAL):
|
||||
return True
|
||||
if 'analysis_report' in self.columns:
|
||||
# Report rows keep QUEUED after handing work to analysis_turn. The
|
||||
# linked turn, not this stored label, is the execution state.
|
||||
for report in self.rows('SELECT status,turn_id FROM analysis_report'):
|
||||
if report['turn_id']:
|
||||
turns=self.rows('SELECT status FROM analysis_turn WHERE id=%s',(report['turn_id'],))
|
||||
if not turns or turns[0]['status'] not in TERMINAL:return True
|
||||
elif report['status'] not in (*TERMINAL,'BLOCKED'):
|
||||
return True
|
||||
if 'source_candidate' in self.columns:
|
||||
return bool(self.rows("SELECT 1 FROM source_candidate WHERE status IN ('QUEUED','RUNNING','IMPORTING') LIMIT 1"))
|
||||
return False
|
||||
|
||||
@contextmanager
|
||||
def frozen(self, expected):
|
||||
"""Short exclusion against writers; never START TRANSACTION after LOCK TABLES.
|
||||
|
||||
All aliases are intentionally avoided while these MySQL locks are held.
|
||||
Busy jobs and a changed release cause rollback without deleting anything.
|
||||
"""
|
||||
if (self.root/'.runtime/retention/paused').exists():raise Busy('Service maintenance is paused')
|
||||
if self.busy():
|
||||
raise Busy('An application job is active')
|
||||
self.conn.autocommit(False)
|
||||
locked = False
|
||||
acquired=None
|
||||
try:
|
||||
self.execute('LOCK TABLES ' + ','.join(ident(t) + ' WRITE' for t in self.tables))
|
||||
locked = True
|
||||
acquired=time.monotonic()
|
||||
if sorted(next(iter(r.values())) for r in self.rows('SHOW TABLES'))!=self.tables:
|
||||
raise Busy('Application schema changed')
|
||||
if self.current() != expected or self.busy():
|
||||
raise Busy('The publication or job state changed')
|
||||
yield
|
||||
self.conn.commit()
|
||||
except BaseException as error:
|
||||
self.conn.rollback()
|
||||
if isinstance(error,pymysql.err.OperationalError) and error.args[0] in (1205,3572):
|
||||
raise Busy('Application tables are in use') from error
|
||||
raise
|
||||
finally:
|
||||
if locked:
|
||||
self.execute('UNLOCK TABLES')
|
||||
self.longest_exclusion=max(self.longest_exclusion,time.monotonic()-acquired)
|
||||
self.conn.autocommit(True)
|
||||
|
||||
def protected_releases(self, current):
|
||||
releases = self.rows('SELECT id,status,parent_release_id,expected_current_release_id FROM data_release')
|
||||
protected = {current}
|
||||
pending = {r['id'] for r in releases if r['status'] != 'PUBLISHED'}
|
||||
protected.update(pending)
|
||||
# Keep candidate baselines and all their ancestors: review/retry must work.
|
||||
while True:
|
||||
more = {r[c] for r in releases if r['id'] in pending for c in ('parent_release_id','expected_current_release_id') if r[c]}
|
||||
if more.issubset(pending):
|
||||
break
|
||||
pending.update(more); protected.update(more)
|
||||
for fk in self.fks:
|
||||
t, c = fk['TABLE_NAME'], fk['COLUMN_NAME']
|
||||
if fk['REFERENCED_TABLE_NAME'] != 'data_release' or t in MUTABLE_RELEASE_REFS or t in OWNED_RELEASE:
|
||||
continue
|
||||
protected.update(r[c] for r in self.rows('SELECT DISTINCT ' + ident(c) + ' FROM ' + ident(t) + ' WHERE ' + ident(c) + ' IS NOT NULL'))
|
||||
# This field is intentionally not a database FK, but is a real user reference.
|
||||
if 'analysis_session' in self.columns:
|
||||
protected.update(r['release_id'] for r in self.rows('SELECT DISTINCT release_id FROM analysis_session'))
|
||||
if 'source_candidate' in self.columns and 'expected_current_release_id' in self.columns['source_candidate']:
|
||||
protected.update(r['expected_current_release_id'] for r in self.rows("SELECT DISTINCT expected_current_release_id FROM source_candidate WHERE status IN ('DISCOVERED','QUEUED','RUNNING') AND expected_current_release_id IS NOT NULL"))
|
||||
if 'source_change_request' in self.columns:
|
||||
cols = [c for c in ('requested_from_release_id','result_release_id') if c in self.columns['source_change_request']]
|
||||
for row in self.rows('SELECT ' + ','.join(map(ident, cols)) + " FROM source_change_request WHERE status IN ('REQUESTED','APPROVED')") if cols else []:
|
||||
protected.update(v for v in row.values() if v)
|
||||
if 'candidate_reconciliation' in self.columns:
|
||||
for row in self.rows('SELECT release_id,baseline_release_id FROM candidate_reconciliation'):
|
||||
if row['release_id'] in pending and row['baseline_release_id']:
|
||||
protected.add(row['baseline_release_id'])
|
||||
return releases, protected
|
||||
|
||||
def release_step(self, current, state, grace, now):
|
||||
releases, protected = self.protected_releases(current)
|
||||
seen = state.setdefault('releases', {})
|
||||
eligible = []
|
||||
ids = {r['id'] for r in releases}
|
||||
for stale in set(seen) - ids:
|
||||
seen.pop(stale)
|
||||
for row in releases:
|
||||
rid = row['id']
|
||||
if rid in protected:
|
||||
seen.pop(rid, None)
|
||||
elif row['status'] == 'PUBLISHED':
|
||||
seen.setdefault(rid, now)
|
||||
if now - seen[rid] >= grace:
|
||||
eligible.append(rid)
|
||||
if not eligible:
|
||||
return 0
|
||||
rid = sorted(eligible, key=lambda x: seen[x])[0]
|
||||
# Clear optional historical links only. Unknown/business references were
|
||||
# included in protected above and are never cascaded away.
|
||||
for fk in self.fks:
|
||||
t,c = fk['TABLE_NAME'],fk['COLUMN_NAME']
|
||||
if fk['REFERENCED_TABLE_NAME'] == 'data_release' and t in MUTABLE_RELEASE_REFS and self.columns[t][c]:
|
||||
self.execute('UPDATE ' + ident(t) + ' SET ' + ident(c) + '=NULL WHERE ' + ident(c) + '=%s', (rid,))
|
||||
if 'data_workflow_operation' in self.columns:
|
||||
for c in ('release_id','expected_current_release_id'):
|
||||
if c in self.columns['data_workflow_operation']:
|
||||
self.execute('UPDATE data_workflow_operation SET ' + ident(c) + '=NULL WHERE ' + ident(c) + '=%s', (rid,))
|
||||
for t in OWNED_RELEASE:
|
||||
if t in self.columns:
|
||||
self.execute('DELETE FROM ' + ident(t) + ' WHERE release_id=%s', (rid,))
|
||||
self.execute('DELETE FROM data_release WHERE id=%s', (rid,))
|
||||
seen.pop(rid, None)
|
||||
return 1
|
||||
|
||||
def build_step(self, state, grace, now, batch):
|
||||
referenced = {r['transform_build_id'] for r in self.rows('SELECT DISTINCT transform_build_id FROM release_member')}
|
||||
builds = self.rows('SELECT id,status,build_key FROM transform_build')
|
||||
seen = state.setdefault('builds', {})
|
||||
ids = {r['id'] for r in builds}
|
||||
for stale in set(seen) - ids:
|
||||
seen.pop(stale)
|
||||
for row in builds:
|
||||
bid = row['id']
|
||||
unknown_reference = any(self.rows('SELECT 1 FROM ' + ident(f['TABLE_NAME']) + ' WHERE ' + ident(f['COLUMN_NAME']) + '=%s LIMIT 1', (bid,))
|
||||
for f in self.fks if f['REFERENCED_TABLE_NAME'] == 'transform_build' and f['TABLE_NAME'] not in (*BUILD_CHILDREN, 'release_member'))
|
||||
if bid in referenced or unknown_reference or row['status'] not in ('COMPLETED','FAILED','CANCELLED','RETIRED'):
|
||||
seen.pop(bid, None)
|
||||
continue
|
||||
seen.setdefault(bid, now)
|
||||
if row['status'] != 'RETIRED' and now - seen[bid] < grace:
|
||||
continue
|
||||
if row['status'] != 'RETIRED':
|
||||
# Free the unique reuse key before deleting chunks. A new import
|
||||
# can build a fresh result, never reuse this partially emptied one.
|
||||
tombstone = hashlib.sha256(('retired:'+bid+':'+row['build_key']).encode()).hexdigest()
|
||||
self.execute("UPDATE transform_build SET status='RETIRED',build_key=%s WHERE id=%s", (tombstone,bid))
|
||||
for t in BUILD_CHILDREN:
|
||||
if t in self.columns:
|
||||
count = self.execute('DELETE FROM ' + ident(t) + ' WHERE build_id=%s LIMIT ' + str(int(batch)), (bid,))
|
||||
if count:
|
||||
return count, 0
|
||||
self.execute('DELETE FROM transform_build WHERE id=%s', (bid,))
|
||||
seen.pop(bid, None)
|
||||
return 0, 1
|
||||
return 0, 0
|
||||
|
||||
def file_referenced(self, path):
|
||||
for table, cols in self.columns.items():
|
||||
for col in cols:
|
||||
if col.endswith('_path') and self.rows('SELECT 1 FROM ' + ident(table) + ' WHERE ' + ident(col) + '=%s LIMIT 1', (str(path),)):
|
||||
return True
|
||||
return False
|
||||
|
||||
def expired(self, limit=10):
|
||||
return self.rows('SELECT * FROM source_upload_draft WHERE expires_at<=TIMESTAMPADD(MINUTE,-30,CURRENT_TIMESTAMP(6)) ORDER BY expires_at LIMIT ' + str(int(limit)))
|
||||
|
||||
def promotion(self, draft):
|
||||
if not draft.get('candidate_id') or not {'source_revision','source_document','source_candidate'}.issubset(self.columns):
|
||||
return None
|
||||
rows = self.rows("SELECT * FROM source_candidate WHERE id=%s AND status='IMPORTED'", (draft['candidate_id'],))
|
||||
if not rows:
|
||||
return None
|
||||
c = rows[0]
|
||||
docs = {d['id']:d for d in self.rows('SELECT id,source_slot FROM source_document')}
|
||||
for rev in self.rows('SELECT source_document_id,archive_path,sha256,parser_config_hash,file_size FROM source_revision WHERE sha256=%s AND parser_config_hash=%s', (c['sha256'],c['parser_config_hash'])):
|
||||
if docs.get(rev['source_document_id'],{}).get('source_slot') == c['source_slot']:
|
||||
p=Path(rev['archive_path'])
|
||||
if safe_file(p,self.root/'runtimedata/archive/source-revisions') and p.stat().st_size==rev['file_size']:
|
||||
return c,rev,p
|
||||
return None
|
||||
|
||||
def clean_owned_scratch(self, br):
|
||||
import psutil
|
||||
server=self.rows('SELECT @@server_uuid AS value')[0]['value']
|
||||
directory=self.root/'.runtime/retention/scratch'
|
||||
removed=0
|
||||
for path in directory.glob('*.json'):
|
||||
if path.is_symlink():continue
|
||||
value=json.loads(path.read_text());name=value.get('database','')
|
||||
if value.get('kind')!='quality-restore-check' or value.get('server')!=server or not value.get('created') or name==self.name or not re.fullmatch(r'quality_restore_\d{14}_[a-f0-9]{8}',name):continue
|
||||
if time.time()-value['createdAt']<300:continue
|
||||
alive=False
|
||||
for pid_key,created_key in [('ownerPid','ownerCreated'),('childPid','childCreated')]:
|
||||
if pid_key not in value:continue
|
||||
try:
|
||||
process=psutil.Process(value[pid_key])
|
||||
if abs(process.create_time()-value[created_key])<.01 and process.is_running() and process.status()!=psutil.STATUS_ZOMBIE:alive=True
|
||||
except psutil.NoSuchProcess:pass
|
||||
except psutil.Error:alive=True
|
||||
if alive or self.rows('SELECT 1 FROM information_schema.processlist WHERE DB=%s LIMIT 1',(name,)):continue
|
||||
self.execute('DROP DATABASE IF EXISTS '+ident(name));path.unlink();removed+=1
|
||||
return removed
|
||||
|
||||
|
||||
def cleanup_drafts(db, current, state, now, br):
|
||||
removed=0; bytes_removed=0
|
||||
# Checksum expensive files outside the brief database exclusion window.
|
||||
for draft in db.expired():
|
||||
path=Path(draft['source_path']); boundary=db.root/'runtimedata/archive/upload-drafts'
|
||||
info=None
|
||||
if safe_file(path,boundary):
|
||||
info=fingerprint(path)
|
||||
if info[2]!=draft['file_size'] or br.sha(path)!=draft['sha256']:
|
||||
continue
|
||||
promotion=db.promotion(draft)
|
||||
if promotion:
|
||||
c,rev,target=promotion
|
||||
target_stamp=fingerprint(target)
|
||||
if br.sha(target)!=rev['sha256']:
|
||||
continue
|
||||
with db.frozen(current):
|
||||
live=db.rows('SELECT * FROM source_upload_draft WHERE id=%s AND expires_at<=TIMESTAMPADD(MINUTE,-30,CURRENT_TIMESTAMP(6))',(draft['id'],))
|
||||
if not live or live[0]!=draft:
|
||||
continue
|
||||
if info and (not safe_file(path,boundary) or fingerprint(path)!=info):
|
||||
continue
|
||||
if promotion and safe_file(target,db.root/'runtimedata/archive/source-revisions') and fingerprint(target)==target_stamp:
|
||||
# The WHERE identity prevents overwriting a source reconfigured
|
||||
# between preflight hashing and acquiring the database exclusion.
|
||||
db.execute("UPDATE source_candidate SET source_path=%s,archive_path=%s WHERE id=%s AND status='IMPORTED' AND source_path=%s AND sha256=%s AND parser_config_hash=%s",
|
||||
(str(target),str(target),c['id'],c['source_path'],c['sha256'],c['parser_config_hash']))
|
||||
db.execute('DELETE FROM source_upload_draft WHERE id=%s',(draft['id'],))
|
||||
if info:
|
||||
with db.frozen(current):
|
||||
if safe_file(path,boundary) and fingerprint(path)==info and not db.file_referenced(path):
|
||||
bytes_removed+=path.stat().st_size
|
||||
path.unlink()
|
||||
removed+=1
|
||||
# Interrupted uploads can leave no database row at all. Their age and every
|
||||
# direct path reference are rechecked under the same exclusion before unlink.
|
||||
boundary=db.root/'runtimedata/archive/upload-drafts'
|
||||
for path in list(boundary.glob('upload-*.xlsx'))[:100]:
|
||||
if not safe_file(path,boundary) or now-path.stat().st_mtime<86400:
|
||||
continue
|
||||
info=fingerprint(path)
|
||||
with db.frozen(current):
|
||||
if fingerprint(path)==info and not db.file_referenced(path):
|
||||
size=path.stat().st_size;path.unlink();bytes_removed+=size
|
||||
return removed,bytes_removed
|
||||
|
||||
|
||||
def run(root, max_batches=30, grace=1800):
|
||||
root=Path(root).resolve();sys.path.append(str(root/'scripts'))
|
||||
import backup_retention as br
|
||||
import backup
|
||||
import backup_verifier
|
||||
import revision_retention
|
||||
if (root/'.runtime/retention/paused').exists():raise Busy('Service maintenance is paused')
|
||||
readiness=urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||
try:
|
||||
with readiness.open('http://127.0.0.1:5281/actuator/health',timeout=2) as response:
|
||||
if json.load(response).get('status')!='UP':raise Busy('Backend is not ready')
|
||||
except (OSError,ValueError) as error:raise Busy('Backend is not ready') from error
|
||||
state_path=root/'.runtime/retention/state.json'
|
||||
state=json.loads(state_path.read_text()) if state_path.exists() else {}
|
||||
result={'checkedAt':time.time(),'releasesRemoved':0,'buildsRemoved':0,'rowsRemoved':0,'draftsRemoved':0,'bytesRemoved':0}
|
||||
with br.lock(root,timeout=0):
|
||||
result['retiredBackupFoldersRemoved']=br.clean_retired(root)
|
||||
result['privateVerifiersRemoved']=backup_verifier.clean_abandoned(root)
|
||||
db=Database(root)
|
||||
try:
|
||||
policy_path=root/'.runtime/retention/config.json'
|
||||
if policy_path.exists():
|
||||
policy=json.loads(policy_path.read_text());accepted=br.accepted_backend(root)
|
||||
if policy['schemaSignature']!=db.schema_signature or not accepted or policy['acceptedJarSha256']!=accepted[1]:
|
||||
raise Busy('Application version changed; retention compatibility review required')
|
||||
result['scratchRemoved']=db.clean_owned_scratch(br)
|
||||
current=db.current()
|
||||
if db.busy():
|
||||
raise Busy('An application job is active')
|
||||
latest=br.latest(root)
|
||||
# Recover a restore-verified candidate left between verification and
|
||||
# pointer publication by a previous crash, without creating a duplicate.
|
||||
if not latest or latest[1].get('queryEvidence',{}).get('releaseId')!=current:
|
||||
candidates=[]
|
||||
for folder in (root/'output/backups').iterdir() if (root/'output/backups').exists() else []:
|
||||
candidate=br.verified(folder,checksums=False)
|
||||
if candidate and candidate.get('queryEvidence',{}).get('releaseId')==current:candidates.append((candidate.get('createdAt',''),folder))
|
||||
if candidates:
|
||||
selected=max(candidates)[1]
|
||||
if br.verified(selected):br.rotate(root,selected);latest=br.latest(root)
|
||||
if latest:
|
||||
state['lastBackupAt']=max(state.get('lastBackupAt',0),datetime.fromisoformat(latest[1]['createdAt']).timestamp())
|
||||
due=not latest or latest[1].get('queryEvidence',{}).get('releaseId')!=current
|
||||
due=due or time.time()-state.get('lastBackupAt',0)>86400
|
||||
due=due or (state.get('dirtySince') and not state.get('builds') and time.time()-state['dirtySince']>300)
|
||||
if due:
|
||||
database_bytes=int(db.rows('SELECT COALESCE(SUM(DATA_LENGTH+INDEX_LENGTH),0) AS size FROM information_schema.tables WHERE TABLE_SCHEMA=%s',(db.name,))[0]['size'])
|
||||
archive_bytes=sum(p.stat().st_size for p in (root/'runtimedata/archive').rglob('*') if p.is_file())
|
||||
required_free=max(10*1024**3,3*database_bytes+2*archive_bytes+5*1024**3)
|
||||
reserves = [root / '.runtime', root / 'output/backups']
|
||||
for reserve in reserves: reserve.mkdir(parents=True, exist_ok=True)
|
||||
if min(shutil.disk_usage(p).free for p in reserves)<required_free:raise Busy('Insufficient reserve for a verified backup; existing backup preserved')
|
||||
if state.get('lastBackupAttemptRelease')==current and time.time()-state.get('lastBackupAttempt',0)<3600:
|
||||
raise Busy('Waiting before retrying a failed backup')
|
||||
state['lastBackupAttempt']=time.time();state['lastBackupAttemptRelease']=current;br.atomic(state_path,state)
|
||||
logging.info('creating consistent backup; previous verified backup remains available')
|
||||
before=br.backup_ids(root)
|
||||
try:
|
||||
folder=backup.create();backup.verify(folder)
|
||||
except BaseException:
|
||||
br.discard_failed_new(root,before)
|
||||
raise
|
||||
if db.current()!=current:
|
||||
raise Busy('Publication changed while backing up; cleanup deferred')
|
||||
latest=br.latest(root) # rotate is shared with manual/UI backup operations
|
||||
if latest is None or latest[0]!=folder:
|
||||
br.rotate(root,folder);latest=br.latest(root)
|
||||
state['lastBackupAt']=time.time();state.pop('dirtySince',None)
|
||||
br.atomic(state_path,state)
|
||||
logging.info('isolated restore verified; latest backup published')
|
||||
if not latest or latest[1].get('queryEvidence',{}).get('releaseId')!=current:
|
||||
raise Busy('No verified backup for the current publication')
|
||||
# Every data-deletion batch rechecks both job state and current pointer.
|
||||
deadline=time.monotonic()+20
|
||||
for _ in range(max_batches):
|
||||
with db.frozen(current):
|
||||
releases=db.release_step(current,state,grace,time.time())
|
||||
rows,builds=db.build_step(state,grace,time.time(),1000)
|
||||
result['releasesRemoved']+=releases;result['rowsRemoved']+=rows;result['buildsRemoved']+=builds
|
||||
if not (releases or rows or builds) or time.monotonic()>deadline or db.longest_exclusion>.5:
|
||||
break
|
||||
time.sleep(.05)
|
||||
drafts,size=cleanup_drafts(db,current,state,time.time(),br)
|
||||
result['draftsRemoved']=drafts;result['bytesRemoved']=size
|
||||
sources=revision_retention.Cleaner(db,br,safe_file,fingerprint,Busy)
|
||||
result.update(sources.collect(current,state,time.time(),grace))
|
||||
result['maxExclusiveSeconds']=round(db.longest_exclusion,4)
|
||||
if any(result[k] for k in ('releasesRemoved','rowsRemoved','buildsRemoved','draftsRemoved','bytesRemoved','sourceRevisionsRemoved','sourceArchiveBytesRemoved')):
|
||||
state.setdefault('dirtySince',time.time())
|
||||
state['lastResult']=result;state.pop('lastError',None)
|
||||
state['consecutiveFailures']=0
|
||||
br.atomic(state_path,state)
|
||||
return result
|
||||
except Busy as error:
|
||||
state['lastDeferred']={'time':time.time(),'reason':str(error)}
|
||||
state['lastResult']=result;br.atomic(state_path,state)
|
||||
raise
|
||||
finally:
|
||||
db.conn.close()
|
||||
|
||||
|
||||
def install(root):
|
||||
root=Path(root).resolve();sys.path.insert(0,str(root/'scripts'));import backup_retention as br
|
||||
python=root/'.runtime/retention-env/bin/python3'
|
||||
if not python.is_file():
|
||||
raise ValueError('Private maintenance runtime missing')
|
||||
files=[Path(__file__).resolve(),root/'scripts/backup_retention.py',root/'scripts/revision_retention.py']
|
||||
db=Database(root)
|
||||
try:
|
||||
accepted=br.accepted_backend(root)
|
||||
if not accepted:raise ValueError('Accepted backend runtime is not registered')
|
||||
br.atomic(root/'.runtime/retention/config.json',{'schemaSignature':db.schema_signature,
|
||||
'acceptedJarSha256':accepted[1],'retirementGraceSeconds':1800,'intervalSeconds':300,
|
||||
'backupIntervalSeconds':86400,'draftExpiryGraceSeconds':1800,'sourceRevisionGraceSeconds':1800})
|
||||
finally:db.conn.close()
|
||||
digest=hashlib.sha256(b''.join(p.read_bytes() for p in files)).hexdigest()
|
||||
version=SUPPORT/'retention'/digest;version.mkdir(parents=True,exist_ok=True,mode=0o700)
|
||||
for p in files:
|
||||
shutil.copy2(p,version/p.name);(version/p.name).chmod(0o600)
|
||||
legacy=root/'.runtime/current-snapshot-20261109'
|
||||
if legacy.is_dir() and not legacy.is_symlink() and (legacy/'manifest.json').is_file():
|
||||
br.atomic(root/'.runtime/retention/legacy.json',{'path':'.runtime/current-snapshot-20261109','manifestSha256':br.sha(legacy/'manifest.json')})
|
||||
logs=SUPPORT/'logs';logs.mkdir(parents=True,exist_ok=True,mode=0o700)
|
||||
plist=Path.home()/'Library/LaunchAgents'/(LABEL+'.plist')
|
||||
value={'Label':LABEL,'ProgramArguments':[str(python),str(version/'retention.py'),'run','--root',str(root)],
|
||||
'WorkingDirectory':str(root),'RunAtLoad':True,'StartInterval':300,'ProcessType':'Background','Nice':10,
|
||||
'LowPriorityIO':True,'ExitTimeOut':300,'EnvironmentVariables':{'PYTHONDONTWRITEBYTECODE':'1'},
|
||||
'StandardOutPath':str(logs/'retention-launchd.log'),'StandardErrorPath':str(logs/'retention-launchd-error.log')}
|
||||
subprocess.run(['/bin/launchctl','bootout','gui/'+str(os.getuid())+'/'+LABEL],capture_output=True)
|
||||
with plist.open('wb') as out:plistlib.dump(value,out)
|
||||
plist.chmod(0o600)
|
||||
subprocess.run(['/bin/launchctl','enable','gui/'+str(os.getuid())+'/'+LABEL],check=True)
|
||||
for attempt in range(50):
|
||||
loaded=subprocess.run(['/bin/launchctl','bootstrap','gui/'+str(os.getuid()),str(plist)],capture_output=True)
|
||||
if loaded.returncode==0:break
|
||||
time.sleep(.2)
|
||||
else:raise RuntimeError('Maintenance registration did not complete')
|
||||
print('Installed five-minute idle maintenance and daily/publication backup checks')
|
||||
|
||||
|
||||
def main():
|
||||
parser=argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action',choices=['run','install','status'])
|
||||
parser.add_argument('--root',required=True)
|
||||
args=parser.parse_args();root=Path(args.root).resolve();sys.path.append(str(root/'scripts'));import backup_retention as br
|
||||
if args.action=='install':
|
||||
with br.lock(root,timeout=0):install(root)
|
||||
return
|
||||
if args.action=='status':
|
||||
p=root/'.runtime/retention/state.json';print(p.read_text() if p.exists() else '{}');return
|
||||
logs=SUPPORT/'logs';logs.mkdir(parents=True,exist_ok=True,mode=0o700)
|
||||
handler=RotatingFileHandler(logs/'retention.log',maxBytes=2*1024*1024,backupCount=2)
|
||||
handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %(message)s'))
|
||||
logging.getLogger().addHandler(handler);logging.getLogger().setLevel(logging.INFO)
|
||||
try:
|
||||
result=run(root);logging.info('maintenance complete %s',json.dumps(result))
|
||||
except (Busy,br.Busy) as error:
|
||||
logging.info('maintenance deferred: %s',str(error))
|
||||
except Exception as error:
|
||||
p=root/'.runtime/retention/state.json';state=json.loads(p.read_text()) if p.exists() else {}
|
||||
state['lastError']={'time':time.time(),'type':type(error).__name__};br.atomic(p,state)
|
||||
state['consecutiveFailures']=min(1000,state.get('consecutiveFailures',0)+1);br.atomic(p,state)
|
||||
logging.error('maintenance failed safely: %s',type(error).__name__)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,351 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Pinned macOS service guardian. Never initializes or imports a database."""
|
||||
import argparse
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
from logging.handlers import RotatingFileHandler
|
||||
import os
|
||||
from pathlib import Path
|
||||
import plistlib
|
||||
import shutil
|
||||
import signal
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
import psutil
|
||||
|
||||
LABEL = 'com.lyunlab.quality-center-service'
|
||||
HOME = Path.home() / 'Library/Application Support/QualityCenterService'
|
||||
CONFIG = HOME / 'service.json'
|
||||
PLIST = Path.home() / 'Library/LaunchAgents' / (LABEL + '.plist')
|
||||
DOMAIN = 'gui/' + str(os.getuid())
|
||||
|
||||
|
||||
def atomic(path, value):
|
||||
path = Path(path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
temporary = path.with_name(path.name + '.tmp-' + str(os.getpid()))
|
||||
with temporary.open('w') as stream:
|
||||
os.chmod(temporary, 0o600)
|
||||
json.dump(value, stream, ensure_ascii=False, indent=2)
|
||||
temporary.replace(path)
|
||||
|
||||
|
||||
def stamp(pid):
|
||||
return subprocess.run(['/bin/ps', '-p', str(pid), '-o', 'lstart='],
|
||||
capture_output=True, text=True).stdout.strip()
|
||||
|
||||
|
||||
def matches(record, command=None):
|
||||
"""True=confirmed identity, False=gone/reused, None=unknown identity."""
|
||||
if not record:
|
||||
return False
|
||||
try:
|
||||
p = psutil.Process(record['pid'])
|
||||
if not p.is_running() or p.status() == psutil.STATUS_ZOMBIE:
|
||||
return False
|
||||
if 'created' in record and abs(p.create_time() - record['created']) > .01:
|
||||
return False
|
||||
if 'started' in record and stamp(p.pid) != record['started']:
|
||||
return False
|
||||
expected = command or record['command']
|
||||
if p.uids().real != os.getuid():
|
||||
return None
|
||||
if Path(p.exe()).resolve() != Path(expected[0]).resolve() or p.cmdline() != expected:
|
||||
return None
|
||||
return True
|
||||
except psutil.NoSuchProcess:
|
||||
return False
|
||||
except (psutil.Error, KeyError, ValueError, OSError):
|
||||
return None
|
||||
|
||||
|
||||
def record(pid, command):
|
||||
return {'pid': pid, 'created': psutil.Process(pid).create_time(),
|
||||
'command': command, 'started': stamp(pid)}
|
||||
|
||||
|
||||
class Guardian:
|
||||
def __init__(self, config, home=HOME):
|
||||
self.config = config
|
||||
self.home = Path(home)
|
||||
self.root = Path(config['root']).resolve()
|
||||
self.runtime = self.root / '.runtime'
|
||||
self.records = {}
|
||||
self.children = {}
|
||||
self.next_start = {'mysql': 0, 'backend': 0}
|
||||
self.restarts = {'mysql': 0, 'backend': 0}
|
||||
self.stopping = False
|
||||
self.opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||
|
||||
def preflight(self):
|
||||
self.value = json.loads((self.runtime / 'local-config.json').read_text())
|
||||
if not self.value.get('databaseReady') or not (self.runtime / 'mysql-data/auto.cnf').is_file():
|
||||
raise RuntimeError('Existing initialized database not confirmed; refusing bootstrap')
|
||||
self.jar = self.runtime / 'backend-builds' / (self.config['jarSha256'] + '.jar')
|
||||
if hashlib.sha256(self.jar.read_bytes()).hexdigest() != self.config['jarSha256']:
|
||||
raise RuntimeError('Accepted backend digest mismatch')
|
||||
for binary in [self.runtime / 'mysql/bin/mysqld', self.runtime / 'jdk/Contents/Home/bin/java']:
|
||||
if not os.access(binary, os.X_OK):
|
||||
raise RuntimeError('Required runtime missing')
|
||||
|
||||
def command(self, role):
|
||||
r = self.runtime
|
||||
if role == 'backend':
|
||||
return [str(r / 'jdk/Contents/Home/bin/java'), '-Xms128m', '-Xmx2048m', '-jar', str(self.jar)]
|
||||
return [str(r / 'mysql/bin/mysqld'), '--no-defaults', '--basedir=' + str(r / 'mysql'),
|
||||
'--datadir=' + str(r / 'mysql-data'), '--port=' + str(self.value['mysqlPort']),
|
||||
'--bind-address=127.0.0.1', '--socket=' + str(r / 'mysql.sock'),
|
||||
'--pid-file=' + str(r / 'mysql-server.pid'),
|
||||
'--log-error=' + str(self.root / 'output/logs/mysql-server.log'), '--mysqlx=OFF',
|
||||
'--lower-case-table-names=2', '--max-connections=60', '--innodb-buffer-pool-size=256M',
|
||||
*(['--skip-log-bin'] if self.value.get('binaryLogEnabled', True) is False else [])]
|
||||
|
||||
def environment(self):
|
||||
v, r = self.value, self.runtime
|
||||
env = os.environ.copy()
|
||||
env.update({
|
||||
'JAVA_HOME': str(r / 'jdk/Contents/Home'),
|
||||
'PATH': str(r / 'jdk/Contents/Home/bin') + ':' + self.config['path'],
|
||||
'MYSQL_URL': f'jdbc:mysql://127.0.0.1:{v["mysqlPort"]}/{v["mysqlDatabase"]}?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC&forceConnectionTimeZoneToSession=true&useSSL=false&allowPublicKeyRetrieval=true',
|
||||
'MYSQL_USER': v['mysqlUser'], 'MYSQL_PASSWORD': v['mysqlPassword'],
|
||||
'MYSQL_LEGACY_TIMESTAMP_OFFSET_SECONDS': str(v.get('legacyTimestampOffsetSeconds', 'UNCONFIRMED')),
|
||||
'QUALITY_BOOTSTRAP_ADMIN_USERNAME': v['adminUsername'],
|
||||
'QUALITY_BOOTSTRAP_ADMIN_PASSWORD': v['adminPassword'],
|
||||
'QUALITY_ARCHIVE_ROOT': str(self.root / 'runtimedata/archive'),
|
||||
'QUALITY_ALLOWED_ROOTS': str(self.root / '质量数据/系统原始数据示例'),
|
||||
'QUALITY_DATA_WORKFLOW_TEST_MODE': 'true',
|
||||
'QUALITY_ANALYSIS_PROTOCOL_V1_ENABLED': str(v.get('analysisProtocolV1Enabled', False)).lower(),
|
||||
'QUALITY_ANALYSIS_AUTONOMOUS_ENABLED': str(v.get('analysisAutonomousEnabled', False)).lower(),
|
||||
'QUALITY_PROJECT_ROOT': str(self.root), 'QUALITY_DEFAULT_SOURCE_YEAR': '2026',
|
||||
'SERVER_ADDRESS': '127.0.0.1', 'PORT': '5281', 'PYTHONDONTWRITEBYTECODE': '1',
|
||||
})
|
||||
log_config = r / 'operations/logback-quality.xml'
|
||||
if log_config.is_file() and not log_config.is_symlink():
|
||||
env['LOGGING_CONFIG'] = str(log_config)
|
||||
env['QUALITY_LOG_DIR'] = str(self.root / 'output/logs')
|
||||
return env
|
||||
|
||||
def port(self, role):
|
||||
return self.value['mysqlPort'] if role == 'mysql' else 5281
|
||||
|
||||
def occupied(self, role):
|
||||
try:
|
||||
with socket.create_connection(('127.0.0.1', self.port(role)), timeout=.5):
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
def db_ready(self):
|
||||
result = subprocess.run([str(self.runtime / 'mysql/bin/mysql'),
|
||||
'--defaults-file=' + str(self.runtime / 'mysql-root.cnf'), '--connect-timeout=2',
|
||||
'--batch', '--skip-column-names', '-e', 'SELECT 1;'],
|
||||
stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=4)
|
||||
return result.returncode == 0 and result.stdout.strip() == b'1'
|
||||
|
||||
def backend_ready(self):
|
||||
try:
|
||||
with self.opener.open('http://127.0.0.1:5281/actuator/health', timeout=2) as response:
|
||||
return json.load(response).get('status') == 'UP'
|
||||
except (OSError, ValueError):
|
||||
return False
|
||||
|
||||
def adopt(self, role):
|
||||
path = self.runtime / (role + '.process.json')
|
||||
if not path.exists():
|
||||
return False
|
||||
info = json.loads(path.read_text())
|
||||
identity = matches(info, self.command(role))
|
||||
if identity is None:
|
||||
raise RuntimeError(role + ': existing process identity cannot be confirmed')
|
||||
if identity:
|
||||
self.records[role] = record(info['pid'], self.command(role))
|
||||
logging.info('%s adopted pid=%s', role, info['pid'])
|
||||
return bool(identity)
|
||||
|
||||
def ensure(self, role):
|
||||
child = self.children.get(role)
|
||||
if child is not None:
|
||||
child.poll() # Reap exited children before checking PID identity.
|
||||
info = self.records.get(role)
|
||||
state = matches(info, self.command(role))
|
||||
if state is None:
|
||||
raise RuntimeError(role + ': process identity is unknown; no replacement started')
|
||||
if state:
|
||||
return True
|
||||
if info:
|
||||
self.records.pop(role)
|
||||
self.next_start[role] = time.monotonic() + 10
|
||||
logging.warning('%s exited; restart scheduled in 10 seconds', role)
|
||||
if time.monotonic() < self.next_start[role]:
|
||||
return False
|
||||
if self.adopt(role):
|
||||
return True
|
||||
if self.occupied(role):
|
||||
raise RuntimeError(role + ': port occupied by an unconfirmed process')
|
||||
command = self.command(role)
|
||||
logs = self.root / 'output/logs'
|
||||
logs.mkdir(parents=True, exist_ok=True)
|
||||
log_name = ('backend-console.log' if role == 'backend' and
|
||||
(self.runtime / 'operations/logback-quality.xml').is_file() else role + '.log')
|
||||
with (logs / log_name).open('ab') as log:
|
||||
child = subprocess.Popen(command, cwd=self.root, env=self.environment(),
|
||||
stdin=subprocess.DEVNULL, stdout=log, stderr=log, start_new_session=True)
|
||||
self.children[role] = child
|
||||
self.records[role] = info = record(child.pid, command)
|
||||
(self.runtime / (role + '.pid')).write_text(str(child.pid))
|
||||
atomic(self.runtime / (role + '.process.json'), {'pid': child.pid, 'started': info['started']})
|
||||
self.restarts[role] += 1
|
||||
logging.info('%s started pid=%s', role, child.pid)
|
||||
return True
|
||||
|
||||
def tick(self):
|
||||
mysql_alive = self.ensure('mysql')
|
||||
ready = mysql_alive and self.db_ready()
|
||||
if ready:
|
||||
self.ensure('backend')
|
||||
atomic(self.home / 'state.json', {'updatedAt': time.time(), 'supervisorPid': os.getpid(),
|
||||
'processes': self.records, 'starts': self.restarts,
|
||||
'databaseReady': bool(ready), 'backendReady': self.backend_ready()})
|
||||
|
||||
def run(self):
|
||||
self.preflight()
|
||||
with (self.runtime / 'quality-service.lock').open('a') as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
for sig in (signal.SIGTERM, signal.SIGINT):
|
||||
signal.signal(sig, lambda *_: setattr(self, 'stopping', True))
|
||||
logging.info('guardian started; accepted backend=%s', self.config['jarSha256'])
|
||||
while not self.stopping:
|
||||
try:
|
||||
self.tick()
|
||||
except Exception as error:
|
||||
# Never log exception arguments: database credentials may be embedded.
|
||||
detail = str(error) if type(error) is RuntimeError else type(error).__name__
|
||||
logging.error('check failed: %s; leaving existing processes intact', detail)
|
||||
atomic(self.home / 'error.json', {'time': time.time(), 'type': type(error).__name__})
|
||||
for _ in range(30):
|
||||
if self.stopping:
|
||||
break
|
||||
time.sleep(.1)
|
||||
logging.info('guardian stopped; service processes remain available for maintenance')
|
||||
|
||||
|
||||
def ctl(*args, check=True):
|
||||
return subprocess.run(['/bin/launchctl', *args], check=check, capture_output=True, text=True)
|
||||
|
||||
|
||||
def install(root, jar, clear_maintenance_pause=True):
|
||||
root, jar = Path(root).resolve(), Path(jar).resolve()
|
||||
cfg = {'root': str(root), 'jarSha256': hashlib.sha256(jar.read_bytes()).hexdigest(),
|
||||
'path': '/opt/homebrew/bin:/usr/local/bin:/Library/Frameworks/Python.framework/Versions/3.13/bin:/usr/bin:/bin:/usr/sbin:/sbin'}
|
||||
Guardian(cfg).preflight()
|
||||
HOME.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
HOME.chmod(0o700)
|
||||
script = Path(__file__).resolve()
|
||||
digest = hashlib.sha256(script.read_bytes()).hexdigest()
|
||||
target = HOME / 'versions' / digest / 'service.py'
|
||||
target.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
shutil.copyfile(script, target)
|
||||
target.chmod(0o600)
|
||||
# Remove only the previous guardian; adopted MySQL/backend processes keep running.
|
||||
ctl('bootout', DOMAIN + '/' + LABEL, check=False)
|
||||
atomic(CONFIG, cfg)
|
||||
logs = HOME / 'logs'
|
||||
logs.mkdir(mode=0o700, exist_ok=True)
|
||||
plist = {'Label': LABEL, 'ProgramArguments': [str(Path(sys.executable).resolve()), str(target), 'run'],
|
||||
'WorkingDirectory': str(HOME), 'RunAtLoad': True, 'KeepAlive': True,
|
||||
'ThrottleInterval': 10, 'ProcessType': 'Background', 'ExitTimeOut': 30,
|
||||
'AbandonProcessGroup': True,
|
||||
'EnvironmentVariables': {'PYTHONDONTWRITEBYTECODE': '1'},
|
||||
'StandardOutPath': str(logs / 'launchd.log'), 'StandardErrorPath': str(logs / 'launchd-error.log')}
|
||||
PLIST.parent.mkdir(exist_ok=True)
|
||||
with PLIST.open('wb') as stream:
|
||||
plistlib.dump(plist, stream)
|
||||
PLIST.chmod(0o600)
|
||||
ctl('enable', DOMAIN + '/' + LABEL)
|
||||
for attempt in range(50):
|
||||
loaded=ctl('bootstrap', DOMAIN, str(PLIST), check=False)
|
||||
if loaded.returncode==0:break
|
||||
time.sleep(.2)
|
||||
else:raise RuntimeError('Service registration did not complete; existing processes preserved')
|
||||
if clear_maintenance_pause:
|
||||
(root/'.runtime/retention/paused').unlink(missing_ok=True)
|
||||
print('Installed login startup and continuous guardian:', PLIST)
|
||||
|
||||
|
||||
def status():
|
||||
loaded = ctl('print', DOMAIN + '/' + LABEL, check=False)
|
||||
state_path = HOME / 'state.json'
|
||||
state = json.loads(state_path.read_text()) if state_path.exists() else {}
|
||||
output = {'launchdLoaded': loaded.returncode == 0, 'stateFresh': time.time() - state.get('updatedAt', 0) < 20,
|
||||
'databaseReady': state.get('databaseReady'), 'backendReady': state.get('backendReady'),
|
||||
'supervisorPid': state.get('supervisorPid'), 'starts': state.get('starts')}
|
||||
output['processes'] = {k: {'pid': v['pid'], 'confirmed': matches(v)} for k, v in state.get('processes', {}).items()}
|
||||
print(json.dumps(output, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
def stop_services():
|
||||
cfg = json.loads(CONFIG.read_text())
|
||||
guardian = Guardian(cfg)
|
||||
guardian.preflight()
|
||||
for role in ('backend', 'mysql'):
|
||||
if not guardian.adopt(role):
|
||||
continue
|
||||
info = guardian.records[role]
|
||||
if matches(info) is not True:
|
||||
raise RuntimeError('Cannot confirm process identity')
|
||||
psutil.Process(info['pid']).terminate()
|
||||
deadline = time.monotonic() + 300
|
||||
while matches(info) is True and time.monotonic() < deadline:
|
||||
time.sleep(.2)
|
||||
if matches(info) is not False:
|
||||
raise RuntimeError('Graceful shutdown incomplete; no forced termination performed')
|
||||
print(role + ' stopped gracefully')
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action', choices=['install', 'run', 'status', 'start', 'pause', 'stop'])
|
||||
parser.add_argument('--root')
|
||||
parser.add_argument('--jar')
|
||||
args = parser.parse_args()
|
||||
if args.action == 'install':
|
||||
if not args.root or not args.jar:
|
||||
parser.error('install requires --root and --jar')
|
||||
install(args.root, args.jar)
|
||||
elif args.action == 'status':
|
||||
status()
|
||||
elif args.action in ('pause', 'stop'):
|
||||
cfg = json.loads(CONFIG.read_text())
|
||||
atomic(Path(cfg['root'])/'.runtime/retention/paused', {'since':time.time()})
|
||||
ctl('disable', DOMAIN + '/' + LABEL)
|
||||
ctl('bootout', DOMAIN + '/' + LABEL, check=False)
|
||||
if args.action == 'stop':
|
||||
stop_services()
|
||||
print('Automatic restart disabled until the start command is used')
|
||||
elif args.action == 'start':
|
||||
cfg = json.loads(CONFIG.read_text())
|
||||
(Path(cfg['root'])/'.runtime/retention/paused').unlink(missing_ok=True)
|
||||
ctl('enable', DOMAIN + '/' + LABEL)
|
||||
if ctl('print', DOMAIN + '/' + LABEL, check=False).returncode:
|
||||
ctl('bootstrap', DOMAIN, str(PLIST))
|
||||
else:
|
||||
ctl('kickstart', DOMAIN + '/' + LABEL)
|
||||
status()
|
||||
else:
|
||||
logs = HOME / 'logs'
|
||||
logs.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
handler = RotatingFileHandler(logs / 'guardian.log', maxBytes=5*1024*1024, backupCount=4)
|
||||
handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %(message)s'))
|
||||
logging.getLogger().addHandler(handler)
|
||||
logging.getLogger().setLevel(logging.INFO)
|
||||
Guardian(json.loads(CONFIG.read_text())).run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Verify the actual JDK, accepted logging libraries, JDBC and cross-language registry."""
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import uuid
|
||||
import zipfile
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0,str(ROOT/'scripts'))
|
||||
import backup
|
||||
import backup_retention as br
|
||||
import managed_test_resources as resources
|
||||
|
||||
|
||||
HARNESS=r'''
|
||||
import java.sql.*;
|
||||
import javax.sql.DataSource;
|
||||
import cn.yunsheng.quality.testsupport.OwnedTestDatabase;
|
||||
public class TestDbProbe {
|
||||
static class DS implements DataSource {
|
||||
public Connection getConnection() throws SQLException {return DriverManager.getConnection(System.getenv("TEST_DB_URL"),"root",System.getenv("TEST_DB_PASSWORD"));}
|
||||
public Connection getConnection(String u,String p)throws SQLException{return DriverManager.getConnection(System.getenv("TEST_DB_URL"),u,p);}
|
||||
public java.io.PrintWriter getLogWriter(){return null;} public void setLogWriter(java.io.PrintWriter w){}
|
||||
public void setLoginTimeout(int t){} public int getLoginTimeout(){return 5;}
|
||||
public java.util.logging.Logger getParentLogger(){return java.util.logging.Logger.getGlobal();}
|
||||
public <T>T unwrap(Class<T> c)throws SQLException{throw new SQLException();} public boolean isWrapperFor(Class<?> c){return false;}
|
||||
}
|
||||
public static void main(String[] a)throws Exception{
|
||||
Class.forName("com.mysql.cj.jdbc.Driver");DS ds=new DS();String name=a[1];
|
||||
if(a[0].equals("normal")){OwnedTestDatabase.create(ds,name);OwnedTestDatabase.drop(ds,name);}
|
||||
else if(a[0].equals("empty")){
|
||||
OwnedTestDatabase.createEmpty(ds,name);
|
||||
try(Connection c=ds.getConnection();Statement s=c.createStatement();ResultSet rs=s.executeQuery("SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='"+name+"'")){
|
||||
rs.next();if(rs.getInt(1)!=0)throw new AssertionError("Fresh schema is not empty");}
|
||||
OwnedTestDatabase.seal(ds,name);OwnedTestDatabase.drop(ds,name);
|
||||
}else OwnedTestDatabase.create(ds,name);
|
||||
System.out.println("PROBE_OK");
|
||||
}
|
||||
}
|
||||
'''
|
||||
|
||||
LOG_PROBE=r'''
|
||||
import ch.qos.logback.classic.LoggerContext;
|
||||
import ch.qos.logback.classic.joran.JoranConfigurator;
|
||||
import ch.qos.logback.core.status.Status;
|
||||
import java.nio.file.*;
|
||||
public class LogProbe {
|
||||
public static void main(String[] args)throws Exception{
|
||||
LoggerContext c=new LoggerContext();c.setMDCAdapter(new ch.qos.logback.classic.util.LogbackMDCAdapter());c.putProperty("QUALITY_LOG_DIR",args[1]);
|
||||
JoranConfigurator config=new JoranConfigurator();config.setContext(c);config.doConfigure(args[0]);
|
||||
for(Status s:c.getStatusManager().getCopyOfStatusList())if(s.getLevel()>=Status.ERROR)throw new AssertionError(s.toString());
|
||||
var logger=c.getLogger("probe");
|
||||
for(int i=0;i<12000;i++)logger.info("{} {}",java.util.UUID.randomUUID(),java.util.UUID.randomUUID());
|
||||
Thread.sleep(1500);
|
||||
for(Status s:c.getStatusManager().getCopyOfStatusList())if(s.getLevel()>=Status.ERROR)throw new AssertionError(s.toString());
|
||||
c.stop();System.out.println("LOG_PROBE_OK");
|
||||
}
|
||||
}
|
||||
'''
|
||||
|
||||
|
||||
class NativeOperationsTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
if not (ROOT / '.runtime/local-config.json').is_file(): raise unittest.SkipTest('Accepted local JDK/MySQL runtime is not configured')
|
||||
cls.temp=tempfile.TemporaryDirectory();cls.build=Path(cls.temp.name).resolve();cls.libs=cls.build/'lib';cls.libs.mkdir()
|
||||
cfg=json.loads((Path.home()/'Library/Application Support/QualityCenterService/service.json').read_text())
|
||||
jar=ROOT/'.runtime/backend-builds'/(cfg['jarSha256']+'.jar')
|
||||
with zipfile.ZipFile(jar) as z:
|
||||
for n in z.namelist():
|
||||
if n.startswith('BOOT-INF/lib/') and any(x in Path(n).name for x in ['logback-','slf4j-api-','spring-boot-4.','spring-core-','mysql-connector-']):
|
||||
(cls.libs/Path(n).name).write_bytes(z.read(n))
|
||||
(cls.build/'TestDbProbe.java').write_text(HARNESS);(cls.build/'LogProbe.java').write_text(LOG_PROBE)
|
||||
cls.java=ROOT/'.runtime/jdk/Contents/Home/bin/java';cls.javac=cls.java.with_name('javac')
|
||||
cls.classes=cls.build/'classes';cls.classes.mkdir()
|
||||
helper=ROOT/'backend/src/test/java/cn/yunsheng/quality/testsupport/OwnedTestDatabase.java'
|
||||
p=subprocess.run([str(cls.javac),'-cp',str(cls.libs/'*'),'-d',str(cls.classes),str(helper),str(cls.build/'TestDbProbe.java'),str(cls.build/'LogProbe.java')],text=True,capture_output=True)
|
||||
if p.returncode:raise RuntimeError(p.stderr)
|
||||
@classmethod
|
||||
def tearDownClass(cls):cls.temp.cleanup()
|
||||
|
||||
def test_java_registration_normal_drop_fresh_schema_and_expired_orphan_cleanup(self):
|
||||
cfg=json.loads((ROOT/'.runtime/local-config.json').read_text())
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root=Path(directory).resolve();br.atomic(root/'.runtime/local-config.json',{'mysqlDatabase':cfg['mysqlDatabase']})
|
||||
env=os.environ.copy();env.update({'TEST_DB_URL':f"jdbc:mysql://127.0.0.1:{cfg['mysqlPort']}/?useSSL=false&allowPublicKeyRetrieval=true",'TEST_DB_PASSWORD':cfg['rootPassword']})
|
||||
names=[]
|
||||
try:
|
||||
for mode in ['normal','empty','orphan']:
|
||||
name='quality_sales_type_test_'+uuid.uuid4().hex;names.append(name)
|
||||
p=subprocess.run([str(self.java),'-cp',str(self.classes)+os.pathsep+str(self.libs/'*'),'TestDbProbe',mode,name],cwd=root,env=env,text=True,capture_output=True,timeout=30)
|
||||
self.assertEqual(p.returncode,0,p.stderr);self.assertIn('PROBE_OK',p.stdout)
|
||||
if mode!='orphan':self.assertIsNone(resources.load(root,name))
|
||||
else:
|
||||
record=resources.load(root,name);self.assertTrue(record['ready'])
|
||||
self.assertFalse(resources.active_process(record['ownerPid'],record['ownerCreated']))
|
||||
result=resources.cleanup(root,backup.client,record['expiresAt']+1)
|
||||
self.assertEqual(result['testDatabasesRemoved'],1);self.assertIsNone(resources.load(root,name))
|
||||
finally:
|
||||
for name in names:backup.client('DROP DATABASE IF EXISTS `'+name+'`;')
|
||||
|
||||
def test_actual_logback_configuration_rolls_and_bounds_files(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root=Path(directory);xml=(ROOT/'deploy/macos/logback-quality.xml').read_text()
|
||||
# Exercise identical policy at a smaller scale, retaining asynchronous-roll headroom.
|
||||
self.assertIn('<maxFileSize>10MB</maxFileSize>',xml);self.assertIn('<totalSizeCap>80MB</totalSizeCap>',xml)
|
||||
config=root/'logback.xml';config.write_text(xml.replace('10MB','10KB').replace('80MB','80KB'))
|
||||
p=subprocess.run([str(self.java),'-cp',str(self.classes)+os.pathsep+str(self.libs/'*'),'LogProbe',str(config),str(root)],text=True,capture_output=True,timeout=30)
|
||||
self.assertEqual(p.returncode,0,p.stderr+p.stdout);self.assertIn('LOG_PROBE_OK',p.stdout)
|
||||
archives=list(root.glob('backend.*.log.gz'));self.assertTrue(archives)
|
||||
self.assertLessEqual(sum(f.stat().st_size for f in archives),90*1024)
|
||||
self.assertLessEqual(sum(f.stat().st_size for f in archives)+(root/'backend.log').stat().st_size,100*1024)
|
||||
self.assertLessEqual((root/'backend.log').stat().st_size,12*1024)
|
||||
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
@@ -0,0 +1,195 @@
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import uuid
|
||||
from unittest.mock import patch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / 'scripts'))
|
||||
import backup
|
||||
import backup_retention as br
|
||||
import log_retention as logs
|
||||
import managed_test_resources as resources
|
||||
spec = importlib.util.spec_from_file_location('quality_operations', Path(__file__).with_name('operations.py'))
|
||||
ops = importlib.util.module_from_spec(spec); spec.loader.exec_module(ops)
|
||||
|
||||
|
||||
class OperationsTests(unittest.TestCase):
|
||||
def sample(self, **extra):
|
||||
return {'at': 200000., 'coreBytes': 12*ops.GIB, 'permanentBytes': 40*ops.GIB,
|
||||
'freeBytes': 500*ops.GIB, 'overdueTestResources': 0, **extra}
|
||||
|
||||
def test_capacity_levels_free_space_failures_and_growth_are_separate(self):
|
||||
self.assertEqual(ops.conditions(self.sample(), ops.DEFAULTS, {}, []), {})
|
||||
self.assertEqual(ops.conditions(self.sample(coreBytes=30*ops.GIB), ops.DEFAULTS, {}, [])['capacity']['severity'], 'warning')
|
||||
self.assertIn('temporary',ops.conditions(self.sample(temporaryBytes=30*ops.GIB),ops.DEFAULTS,{},[]))
|
||||
self.assertIn('artifacts',ops.conditions(self.sample(artifactBytes=50*ops.GIB),ops.DEFAULTS,{},[]))
|
||||
found = ops.conditions(self.sample(coreBytes=50*ops.GIB, freeBytes=19*ops.GIB, overdueTestResources=1),
|
||||
ops.DEFAULTS, {'consecutiveFailures': 3}, [])
|
||||
self.assertEqual(set(found), {'capacity', 'disk', 'maintenance', 'tests'})
|
||||
old = self.sample(at=200000-86400, permanentBytes=30*ops.GIB)
|
||||
self.assertIn('growth', ops.conditions(self.sample(), ops.DEFAULTS, {}, [old]))
|
||||
self.assertNotIn('maintenance', ops.conditions(self.sample(), ops.DEFAULTS, {'lastDeferred': {'reason': 'active'}}, []))
|
||||
old = self.sample(at=200000-86400)
|
||||
self.assertIn('growth',ops.conditions(self.sample(coreBytes=15*ops.GIB,permanentBytes=43*ops.GIB),ops.DEFAULTS,{},[old]))
|
||||
|
||||
def test_monitor_failure_notifies_after_three_attempts_without_repeated_noise(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
root=Path(d).resolve();sent=[]
|
||||
def notify(*_):sent.append(True);return {'submitted':True}
|
||||
for _ in range(4):ops.failed_run(root,RuntimeError('not logged'),notify)
|
||||
self.assertEqual(len(sent),1)
|
||||
self.assertEqual(json.loads((root/'.runtime/operations/failure.json').read_text())['count'],4)
|
||||
|
||||
def test_alert_deduplication_escalation_recovery_and_failed_delivery_retry(self):
|
||||
sent = []
|
||||
def notify(title, message): sent.append(message); return {'submitted': True}
|
||||
warning = {'capacity': {'severity': 'warning', 'message': 'warning'}}
|
||||
first, events = ops.dispatch(warning, {}, 100, ops.DEFAULTS, notify)
|
||||
second, events = ops.dispatch(warning, first, 200, ops.DEFAULTS, notify)
|
||||
self.assertEqual(len(sent), 1); self.assertFalse(events)
|
||||
critical = {'capacity': {'severity': 'critical', 'message': 'critical'}}
|
||||
third, _ = ops.dispatch(critical, second, 300, ops.DEFAULTS, notify)
|
||||
self.assertEqual(len(sent), 2)
|
||||
_, events = ops.dispatch({}, third, 400, ops.DEFAULTS, notify)
|
||||
self.assertEqual(events[0]['kind'], 'resolved')
|
||||
failed, _ = ops.dispatch(warning, {}, 500, ops.DEFAULTS, lambda *_: {'submitted': False})
|
||||
ops.dispatch(warning, failed, 600, ops.DEFAULTS, notify)
|
||||
self.assertEqual(len(sent), 3)
|
||||
|
||||
def test_registered_and_private_temporary_storage_is_not_permanent_growth(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
root = Path(d).resolve(); support = root / 'support'
|
||||
name = 'quality_retention_test_' + uuid.uuid4().hex[:12]
|
||||
for path in [root/'.runtime/mysql-data/current/data.ibd', root/'.runtime/mysql-data'/name/'sample.ibd',
|
||||
root/'output/backups/current/file.gz', root/'runtimedata/archive/source.xlsx']:
|
||||
path.parent.mkdir(parents=True, exist_ok=True); path.write_bytes(b'x'*8192)
|
||||
br.atomic(resources.registry(root)/(name+'.json'), {'kind':'quality-owned-test-db','root':str(root),
|
||||
'database':name,'token':'1'*32,'created':True,'expiresAt':time.time()+86400})
|
||||
private=root/'.runtime/retention/verify-fixture';private.mkdir(parents=True)
|
||||
(private/'data').write_bytes(b'x'*16384)
|
||||
br.atomic(private/'owner.json',{'kind':'private-quality-restore','directory':str(private)})
|
||||
sample=ops.snapshot(root,support)
|
||||
self.assertGreater(sample['registeredTestBytes'],0); self.assertGreater(sample['privateVerifierBytes'],0)
|
||||
self.assertLess(sample['coreBytes'],sample['mysqlBytes']+sample['backupBytes']+sample['archiveBytes'])
|
||||
|
||||
def test_monitor_still_collects_when_housekeeping_is_deferred_and_history_is_bounded(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
root=Path(d).resolve();base=root/'.runtime/operations';base.mkdir(parents=True)
|
||||
br.atomic(base/'policy.json',ops.DEFAULTS)
|
||||
history=[self.sample(at=time.time()-i*100) for i in range(450)]
|
||||
br.atomic(base/'samples.json',history)
|
||||
with patch.object(ops,'snapshot',return_value=self.sample(at=time.time())), \
|
||||
patch.object(ops,'janitor',return_value={'deferred':'Application jobs are active'}):
|
||||
result=ops.run(root,root/'support',lambda *_:{'submitted':True})
|
||||
self.assertIn('sample',result);self.assertIn('deferred',result['cleanup'])
|
||||
self.assertLessEqual(len(json.loads((base/'samples.json').read_text())),330)
|
||||
|
||||
|
||||
class LogRetentionTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp=tempfile.TemporaryDirectory();self.root=Path(self.temp.name).resolve();self.support=self.root/'support'
|
||||
def tearDown(self):self.temp.cleanup()
|
||||
def file(self,name,age=0,size=100):
|
||||
p=self.root/'output/logs'/name;p.parent.mkdir(parents=True,exist_ok=True);p.write_bytes(b'line\n'*(size//5))
|
||||
os.utime(p,(time.time()-age,time.time()-age));return p
|
||||
|
||||
def test_old_closed_logs_removed_but_active_symlinks_and_business_files_preserved(self):
|
||||
old=self.file('old.log',age=8*86400);active=self.file('backend.log',age=8*86400)
|
||||
other=self.file('worker.log',age=8*86400);outside=self.root/'important.xlsx';outside.write_bytes(b'keep')
|
||||
link=old.with_name('link.log');link.symlink_to(outside)
|
||||
result=logs.retain(self.root,self.support,open_paths={str(other)})
|
||||
self.assertEqual(result['logsRemoved'],1);self.assertFalse(old.exists())
|
||||
self.assertTrue(active.exists());self.assertTrue(other.exists());self.assertTrue(outside.exists());self.assertTrue(link.is_symlink())
|
||||
|
||||
def test_per_class_cap_prunes_old_archives_and_bounds_a_closed_console_tail(self):
|
||||
old=self.file('worker.log.1',age=100,size=100);new=self.file('worker.log',age=0,size=100)
|
||||
logs.retain(self.root,self.support,open_paths=set(),max_bytes=150)
|
||||
self.assertFalse(old.exists());self.assertTrue(new.exists())
|
||||
big=self.file('huge.log',size=1000)
|
||||
logs.retain(self.root,self.support,open_paths=set(),max_bytes=100)
|
||||
self.assertLessEqual(big.stat().st_size,100)
|
||||
|
||||
def test_mysql_rotation_uses_reopen_and_recovers_after_interruption(self):
|
||||
source=self.file('mysql-server.log',age=86400,size=100);original=source.read_bytes()
|
||||
with self.assertRaises(RuntimeError):
|
||||
logs.rotate_mysql(self.root,lambda *_:(_ for _ in ()).throw(RuntimeError('interrupted')),br)
|
||||
self.assertFalse(source.exists());self.assertTrue((self.root/'.runtime/operations/mysql-log-rotation.json').exists())
|
||||
calls=[]
|
||||
def reopen(sql,_):calls.append(sql);source.write_bytes(b'new log')
|
||||
self.assertEqual(logs.rotate_mysql(self.root,reopen,br),1)
|
||||
self.assertEqual(calls,['FLUSH ERROR LOGS;'])
|
||||
archived=list(source.parent.glob('mysql-server.roll-*.log'));self.assertEqual(archived[0].read_bytes(),original)
|
||||
self.assertEqual(source.read_bytes(),b'new log')
|
||||
|
||||
|
||||
class ManagedResourceTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
if not (ROOT / '.runtime/local-config.json').is_file(): raise unittest.SkipTest('Local MySQL test runtime is not configured')
|
||||
def setUp(self):
|
||||
self.temp=tempfile.TemporaryDirectory();self.root=Path(self.temp.name).resolve()
|
||||
cfg=json.loads((ROOT/'.runtime/local-config.json').read_text())
|
||||
br.atomic(self.root/'.runtime/local-config.json',{'mysqlDatabase':cfg['mysqlDatabase']})
|
||||
self.names=[];self.users=[]
|
||||
def tearDown(self):
|
||||
for n in self.names:backup.client('DROP DATABASE IF EXISTS `'+n+'`;')
|
||||
for u in self.users:backup.client("DROP USER IF EXISTS '"+u+"'@'127.0.0.1';")
|
||||
self.temp.cleanup()
|
||||
def create(self,empty=False):
|
||||
name='quality_retention_test_'+uuid.uuid4().hex[:12];self.names.append(name)
|
||||
resources.create(self.root,name,backup.client,empty=empty);return name
|
||||
|
||||
def test_successful_create_and_drop_unregisters_only_its_schema(self):
|
||||
name=self.create();self.assertTrue(resources.load(self.root,name)['ready'])
|
||||
resources.drop(self.root,name,backup.client);self.assertIsNone(resources.load(self.root,name))
|
||||
|
||||
def test_expired_dead_owner_is_cleaned_but_live_owner_is_protected(self):
|
||||
name=self.create();value=resources.load(self.root,name);now=value['expiresAt']+1
|
||||
self.assertEqual(resources.cleanup(self.root,backup.client,now)['testDatabasesRemoved'],0)
|
||||
with patch.object(resources,'active_process',return_value=False):
|
||||
self.assertEqual(resources.cleanup(self.root,backup.client,now)['testDatabasesRemoved'],1)
|
||||
self.assertIsNone(resources.load(self.root,name))
|
||||
|
||||
def test_unregistered_schema_wrong_server_and_wrong_marker_are_never_dropped(self):
|
||||
name=self.create();value=resources.load(self.root,name);now=value['expiresAt']+1
|
||||
value['serverUuid']='different';br.atomic(resources.registry(self.root)/(name+'.json'),value)
|
||||
with patch.object(resources,'active_process',return_value=False):
|
||||
self.assertEqual(resources.cleanup(self.root,backup.client,now)['testDatabasesRemoved'],0)
|
||||
with self.assertRaises(ValueError):resources.drop(self.root,name,backup.client)
|
||||
value['serverUuid']=backup.client('SELECT @@server_uuid;').strip();br.atomic(resources.registry(self.root)/(name+'.json'),value)
|
||||
backup.client('UPDATE `'+name+'`.`'+resources.MARKER+"` SET token='different';")
|
||||
with patch.object(resources,'active_process',return_value=False):
|
||||
self.assertEqual(resources.cleanup(self.root,backup.client,now)['testDatabasesRemoved'],0)
|
||||
with self.assertRaises(ValueError):resources.create(self.root,'quality_platform',backup.client)
|
||||
|
||||
def test_empty_schema_stays_empty_until_migrations_finish_then_seals(self):
|
||||
name=self.create(empty=True)
|
||||
self.assertEqual(backup.client('SHOW TABLES;',name).strip(),'')
|
||||
backup.client('CREATE TABLE business_data(id INT);',name)
|
||||
resources.seal(self.root,name,backup.client)
|
||||
self.assertTrue(resources.marker_matches(resources.load(self.root,name),backup.client))
|
||||
resources.drop(self.root,name,backup.client)
|
||||
|
||||
def test_empty_failed_schema_can_be_dropped_by_its_current_owner(self):
|
||||
name=self.create(empty=True);resources.drop(self.root,name,backup.client)
|
||||
self.assertIsNone(resources.load(self.root,name))
|
||||
|
||||
def test_registered_test_account_is_cleaned_after_expired_schema(self):
|
||||
name=self.create();user='qverify_'+uuid.uuid4().hex[:12];self.users.append(user)
|
||||
backup.client("CREATE USER '"+user+"'@'127.0.0.1' IDENTIFIED BY 'disposable-test-account';")
|
||||
backup.client('GRANT ALL ON `'+name+"`.* TO '"+user+"'@'127.0.0.1';")
|
||||
resources.attach_user(self.root,name,user)
|
||||
with patch.object(resources,'active_process',return_value=False):
|
||||
result=resources.cleanup(self.root,backup.client,resources.load(self.root,name)['expiresAt']+1)
|
||||
self.assertEqual(result['testDatabasesRemoved'],1);self.assertEqual(result['testAccountsRemoved'],1)
|
||||
self.assertIsNone(resources.load(self.root,name))
|
||||
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
@@ -0,0 +1,305 @@
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import tarfile
|
||||
import gzip
|
||||
from contextlib import contextmanager
|
||||
import unittest
|
||||
import uuid
|
||||
from unittest.mock import patch
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0,str(ROOT/'scripts'))
|
||||
import backup_retention as br
|
||||
spec=importlib.util.spec_from_file_location('quality_retention',Path(__file__).with_name('retention.py'))
|
||||
ret=importlib.util.module_from_spec(spec);spec.loader.exec_module(ret)
|
||||
|
||||
|
||||
def good_backup(root, suffix, created='2026-11-09T01:00:00+00:00'):
|
||||
folder=root/'output/backups'/('20261109T010000Z-'+suffix)
|
||||
folder.mkdir(parents=True)
|
||||
m={'formatVersion':3,'status':'ISOLATED_RESTORE_VERIFIED','createdAt':created,
|
||||
'restoreVerifiedAt':created,'queryEvidence':{'releaseId':'current'},
|
||||
'verification':{k:True for k in br.CHECKS}}
|
||||
for name,key in br.ARTIFACTS.items():
|
||||
(folder/name).write_bytes(name.encode());m[key]=br.sha(folder/name)
|
||||
br.atomic(folder/'manifest.json',m)
|
||||
return folder.resolve()
|
||||
|
||||
|
||||
class BackupRotationTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp=tempfile.TemporaryDirectory();self.root=Path(self.temp.name)
|
||||
def tearDown(self):self.temp.cleanup()
|
||||
|
||||
def test_failed_or_corrupted_new_backup_keeps_previous(self):
|
||||
old=good_backup(self.root,'11111111');br.rotate(self.root,old)
|
||||
new=good_backup(self.root,'22222222','2026-11-09T02:00:00+00:00')
|
||||
(new/'database.sql.gz').write_bytes(b'corrupted')
|
||||
with self.assertRaises(ValueError):br.rotate(self.root,new)
|
||||
self.assertTrue(old.is_dir());self.assertEqual(br.latest(self.root)[0],old)
|
||||
|
||||
def test_rotation_keeps_one_and_verifying_older_never_rolls_back_latest(self):
|
||||
old=good_backup(self.root,'11111111');br.rotate(self.root,old)
|
||||
new=good_backup(self.root,'22222222','2026-11-09T02:00:00+00:00');br.rotate(self.root,new)
|
||||
self.assertFalse(old.exists());self.assertEqual(br.latest(self.root)[0],new)
|
||||
older=good_backup(self.root,'33333333');br.rotate(self.root,older)
|
||||
self.assertEqual(br.latest(self.root)[0],new);self.assertFalse(older.exists())
|
||||
|
||||
def test_unknown_and_active_incomplete_folders_are_preserved(self):
|
||||
old=good_backup(self.root,'11111111');unknown=old.parent/'unrelated';unknown.mkdir()
|
||||
incomplete=old.parent/'20261109T010000Z-abcdefab';incomplete.mkdir()
|
||||
br.atomic(incomplete/'.owner.json',{'kind':'quality-backup','root':str(self.root),'id':incomplete.name})
|
||||
new=good_backup(self.root,'22222222','2026-11-09T02:00:00+00:00');br.rotate(self.root,new)
|
||||
self.assertTrue(unknown.exists());self.assertTrue(incomplete.exists())
|
||||
|
||||
def test_shared_lock_excludes_another_process(self):
|
||||
code='import sys;from pathlib import Path;sys.path.insert(0,sys.argv[1]);import backup_retention as b\ntry:\n with b.lock(Path(sys.argv[2]),timeout=0):pass\nexcept b.Busy:sys.exit(17)'
|
||||
with br.lock(self.root):
|
||||
child=subprocess.run([sys.executable,'-c',code,str(ROOT/'scripts'),str(self.root)],capture_output=True)
|
||||
self.assertEqual(child.returncode,17)
|
||||
|
||||
def test_managed_backup_root_cannot_be_redirected_outside_project(self):
|
||||
with tempfile.TemporaryDirectory() as external:
|
||||
folder=good_backup(Path(external),'11111111')
|
||||
(self.root/'output').mkdir();(self.root/'output/backups').symlink_to(folder.parent,target_is_directory=True)
|
||||
with self.assertRaises(ValueError):br.rotate(self.root,folder)
|
||||
self.assertTrue(folder.is_dir())
|
||||
|
||||
def test_interrupted_old_backup_removal_can_resume_without_touching_latest(self):
|
||||
old=good_backup(self.root,'11111111');br.rotate(self.root,old)
|
||||
new=good_backup(self.root,'22222222','2026-11-09T02:00:00+00:00')
|
||||
with patch.object(br.shutil,'rmtree',side_effect=OSError('simulated removal failure')):
|
||||
br.rotate(self.root,new)
|
||||
self.assertEqual(br.latest(self.root)[0],new)
|
||||
self.assertEqual(len(list((self.root/'.runtime/retention/retired').glob('*.json'))),1)
|
||||
br.clean_retired(self.root)
|
||||
self.assertFalse(list((self.root/'.runtime/retention/retired').glob('*.json')))
|
||||
self.assertTrue(new.is_dir())
|
||||
|
||||
|
||||
class DatabaseRetentionTests(unittest.TestCase):
|
||||
"""Real MySQL locking/FK tests in a newly owned, tiny disposable database."""
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
if not (ROOT / '.runtime/local-config.json').is_file(): raise unittest.SkipTest('Local MySQL test runtime is not configured')
|
||||
cfg=json.loads((ROOT/'.runtime/local-config.json').read_text())
|
||||
cls.admin=ret.pymysql.connect(unix_socket=str(ROOT/'.runtime/mysql.sock'),user='root',password=cfg['rootPassword'],autocommit=True)
|
||||
cls.name='quality_retention_test_'+uuid.uuid4().hex[:12]
|
||||
with cls.admin.cursor() as c:
|
||||
c.execute('CREATE DATABASE '+ret.ident(cls.name))
|
||||
c.execute('USE '+ret.ident(cls.name))
|
||||
ddl=[
|
||||
'CREATE TABLE data_release(id varchar(36) PRIMARY KEY,status varchar(24),parent_release_id varchar(36),expected_current_release_id varchar(36),FOREIGN KEY(parent_release_id) REFERENCES data_release(id),FOREIGN KEY(expected_current_release_id) REFERENCES data_release(id))',
|
||||
'CREATE TABLE current_release(dataset_id varchar(20) PRIMARY KEY,release_id varchar(36),FOREIGN KEY(release_id) REFERENCES data_release(id))',
|
||||
'CREATE TABLE transform_build(id varchar(36) PRIMARY KEY,status varchar(24),build_key char(64) UNIQUE)',
|
||||
'CREATE TABLE release_member(release_id varchar(36),transform_build_id varchar(36),FOREIGN KEY(release_id) REFERENCES data_release(id),FOREIGN KEY(transform_build_id) REFERENCES transform_build(id))',
|
||||
'CREATE TABLE fact_black(id int PRIMARY KEY,build_id varchar(36),payload varchar(20),FOREIGN KEY(build_id) REFERENCES transform_build(id))',
|
||||
'CREATE TABLE unknown_build_use(id int PRIMARY KEY,build_id varchar(36),FOREIGN KEY(build_id) REFERENCES transform_build(id))',
|
||||
'CREATE TABLE task(id int PRIMARY KEY,status varchar(24),next_retry_at datetime,retry_count int,max_retries int)',
|
||||
'CREATE TABLE import_run(id int PRIMARY KEY,status varchar(24),expected_current_release_id varchar(36),result_release_id varchar(36),FOREIGN KEY(expected_current_release_id) REFERENCES data_release(id),FOREIGN KEY(result_release_id) REFERENCES data_release(id))',
|
||||
'CREATE TABLE release_action(id int PRIMARY KEY,release_id varchar(36),expected_current_release_id varchar(36),FOREIGN KEY(release_id) REFERENCES data_release(id),FOREIGN KEY(expected_current_release_id) REFERENCES data_release(id))',
|
||||
'CREATE TABLE candidate_reconciliation(id int PRIMARY KEY,release_id varchar(36),baseline_release_id varchar(36),FOREIGN KEY(release_id) REFERENCES data_release(id),FOREIGN KEY(baseline_release_id) REFERENCES data_release(id))',
|
||||
'CREATE TABLE analysis_session(id int PRIMARY KEY,release_id varchar(36))',
|
||||
'CREATE TABLE saved_business_object(id int PRIMARY KEY,release_id varchar(36),FOREIGN KEY(release_id) REFERENCES data_release(id))',
|
||||
'CREATE TABLE source_document(id varchar(36) PRIMARY KEY,source_slot varchar(100))',
|
||||
'CREATE TABLE source_revision(id varchar(36) PRIMARY KEY,source_document_id varchar(36),archive_path varchar(1000),sha256 char(64),parser_config_hash varchar(64),file_size bigint)',
|
||||
'CREATE TABLE source_candidate(id varchar(36) PRIMARY KEY,status varchar(24),source_slot varchar(100),source_path varchar(1000),archive_path varchar(1000),sha256 char(64),parser_config_hash varchar(64),expected_current_release_id varchar(36))',
|
||||
'CREATE TABLE source_upload_draft(id varchar(36) PRIMARY KEY,source_file varchar(100),source_path varchar(1000),sha256 char(64),file_size bigint,candidate_id varchar(36),created_at datetime,expires_at datetime)']
|
||||
for statement in ddl:c.execute(statement+' ENGINE=InnoDB')
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
with cls.admin.cursor() as c:c.execute('DROP DATABASE '+ret.ident(cls.name))
|
||||
cls.admin.close()
|
||||
|
||||
def setUp(self):
|
||||
self.temp=tempfile.TemporaryDirectory();self.root=Path(self.temp.name)
|
||||
self.db=ret.Database(ROOT,self.name);self.db.root=self.root
|
||||
self.db.execute('SET FOREIGN_KEY_CHECKS=0')
|
||||
for table in self.db.tables:self.db.execute('TRUNCATE TABLE '+ret.ident(table))
|
||||
self.db.execute('SET FOREIGN_KEY_CHECKS=1')
|
||||
self.db.execute("INSERT INTO data_release VALUES('old','PUBLISHED',NULL,NULL),('current','PUBLISHED','old','old')")
|
||||
self.db.execute("INSERT INTO current_release VALUES('quality','current')")
|
||||
self.db.execute("INSERT INTO transform_build VALUES('shared','COMPLETED',%s),('obsolete','COMPLETED',%s)",('1'*64,'2'*64))
|
||||
self.db.execute("INSERT INTO release_member VALUES('current','shared'),('old','shared'),('old','obsolete')")
|
||||
self.db.execute("INSERT INTO fact_black VALUES(1,'shared','current data'),(2,'obsolete','old data'),(3,'obsolete','old data')")
|
||||
|
||||
def tearDown(self):self.db.conn.close();self.temp.cleanup()
|
||||
|
||||
def retire_old(self):
|
||||
state={'releases':{'old':1}}
|
||||
with self.db.frozen('current'):self.assertEqual(self.db.release_step('current',state,1800,4000),1)
|
||||
|
||||
def test_running_queued_retry_and_unknown_tasks_prevent_cleanup(self):
|
||||
for status,retry in [('RUNNING',False),('QUEUED',False),('FAILED',True),('FUTURE_STATE',False)]:
|
||||
self.db.execute('DELETE FROM task')
|
||||
self.db.execute('INSERT INTO task VALUES(1,%s,%s,0,3)',(status,datetime_from_future() if retry else None))
|
||||
with self.assertRaises(ret.Busy):
|
||||
with self.db.frozen('current'):self.fail('busy job was ignored')
|
||||
self.assertEqual(len(self.db.rows('SELECT * FROM data_release')),2)
|
||||
|
||||
def test_pending_candidate_and_saved_business_references_are_protected(self):
|
||||
self.db.execute("INSERT INTO data_release VALUES('candidate','CANDIDATE','old','old')")
|
||||
with self.db.frozen('current'):
|
||||
self.assertEqual(self.db.release_step('current',{'releases':{'old':1}},1800,4000),0)
|
||||
self.db.execute("DELETE FROM data_release WHERE id='candidate'")
|
||||
self.db.execute("INSERT INTO saved_business_object VALUES(1,'old')")
|
||||
with self.db.frozen('current'):
|
||||
self.assertEqual(self.db.release_step('current',{'releases':{'old':1}},1800,4000),0)
|
||||
|
||||
def test_analysis_session_reference_is_protected_without_database_fk(self):
|
||||
self.db.execute("INSERT INTO analysis_session VALUES(1,'old')")
|
||||
with self.db.frozen('current'):
|
||||
self.assertEqual(self.db.release_step('current',{'releases':{'old':1}},1800,4000),0)
|
||||
|
||||
def test_pending_source_configuration_preserves_its_old_baseline(self):
|
||||
self.db.execute("INSERT INTO source_candidate VALUES('pending','DISCOVERED','slot','path',NULL,%s,'legacy','old')",('3'*64,))
|
||||
with self.db.frozen('current'):
|
||||
self.assertEqual(self.db.release_step('current',{'releases':{'old':1}},1800,4000),0)
|
||||
|
||||
def test_unknown_business_build_reference_is_preserved(self):
|
||||
self.retire_old();self.db.execute("INSERT INTO unknown_build_use VALUES(1,'obsolete')")
|
||||
with self.db.frozen('current'):
|
||||
self.assertEqual(self.db.build_step({'builds':{'obsolete':1}},1800,4000,1),(0,0))
|
||||
self.assertEqual(self.db.rows("SELECT status FROM transform_build WHERE id='obsolete'")[0]['status'],'COMPLETED')
|
||||
|
||||
def test_changed_publication_and_grace_period_prevent_cleanup(self):
|
||||
with self.assertRaises(ret.Busy):
|
||||
with self.db.frozen('old'):pass
|
||||
with self.db.frozen('current'):
|
||||
self.assertEqual(self.db.release_step('current',{'releases':{'old':3900}},1800,4000),0)
|
||||
|
||||
def test_old_release_removed_but_current_membership_and_rows_preserved(self):
|
||||
self.db.execute("INSERT INTO import_run VALUES(1,'COMPLETED','old','old')")
|
||||
self.db.execute("INSERT INTO release_action VALUES(1,'old',NULL)")
|
||||
self.db.execute("INSERT INTO candidate_reconciliation VALUES(1,'current','old')")
|
||||
self.retire_old()
|
||||
self.assertEqual(self.db.rows("SELECT payload FROM fact_black WHERE build_id='shared'")[0]['payload'],'current data')
|
||||
self.assertEqual(self.db.rows("SELECT * FROM release_member"),[{'release_id':'current','transform_build_id':'shared'}])
|
||||
self.assertIsNone(self.db.rows('SELECT * FROM import_run')[0]['expected_current_release_id'])
|
||||
self.assertIsNone(self.db.rows("SELECT * FROM data_release WHERE id='current'")[0]['parent_release_id'])
|
||||
|
||||
def test_shared_build_is_preserved_and_partial_retirement_cannot_be_reused(self):
|
||||
self.retire_old();state={'builds':{'obsolete':1}}
|
||||
with self.db.frozen('current'):
|
||||
self.assertEqual(self.db.build_step(state,1800,4000,1),(1,0))
|
||||
old=self.db.rows("SELECT * FROM transform_build WHERE id='obsolete'")[0]
|
||||
self.assertEqual(old['status'],'RETIRED');self.assertNotEqual(old['build_key'],'2'*64)
|
||||
self.db.execute("INSERT INTO transform_build VALUES('fresh','COMPLETED',%s)",('2'*64,))
|
||||
for _ in range(4):
|
||||
with self.db.frozen('current'):self.db.build_step(state,1800,4000,1)
|
||||
self.assertFalse(self.db.rows("SELECT * FROM transform_build WHERE id='obsolete'"))
|
||||
self.assertTrue(self.db.rows("SELECT * FROM transform_build WHERE id='shared'"))
|
||||
self.assertTrue(self.db.rows("SELECT * FROM transform_build WHERE id='fresh'"))
|
||||
|
||||
def add_draft(self,candidate=None,where=None,age=25):
|
||||
path=where or self.root/'runtimedata/archive/upload-drafts/upload-fixture.xlsx'
|
||||
path.parent.mkdir(parents=True,exist_ok=True);path.write_bytes(b'workbook bytes')
|
||||
self.db.execute('INSERT INTO source_upload_draft VALUES(%s,%s,%s,%s,%s,%s,TIMESTAMPADD(HOUR,-30,CURRENT_TIMESTAMP),TIMESTAMPADD(HOUR,%s,CURRENT_TIMESTAMP))',('draft','input.xlsx',str(path),br.sha(path),path.stat().st_size,candidate,-age))
|
||||
return path
|
||||
|
||||
def test_expired_unbound_upload_removed_but_fresh_preview_is_kept(self):
|
||||
path=self.add_draft()
|
||||
count,size=ret.cleanup_drafts(self.db,'current',{},time.time(),br)
|
||||
self.assertEqual(count,1);self.assertGreater(size,0);self.assertFalse(path.exists())
|
||||
self.add_draft(age=0)
|
||||
self.assertEqual(ret.cleanup_drafts(self.db,'current',{},time.time(),br),(0,0))
|
||||
|
||||
def test_upload_still_referenced_by_pending_candidate_is_kept(self):
|
||||
path=self.add_draft(candidate='pending')
|
||||
self.db.execute("INSERT INTO source_candidate VALUES('pending','DISCOVERED','slot',%s,NULL,%s,'legacy',NULL)",(str(path),br.sha(path)))
|
||||
self.assertEqual(ret.cleanup_drafts(self.db,'current',{},time.time(),br)[0],1)
|
||||
self.assertTrue(path.exists())
|
||||
|
||||
def test_imported_upload_is_rebound_to_verified_archive_before_cleanup(self):
|
||||
path=self.add_draft(candidate='imported');archive=self.root/'runtimedata/archive/source-revisions/formal.xlsx';archive.parent.mkdir();archive.write_bytes(path.read_bytes())
|
||||
self.db.execute("INSERT INTO source_document VALUES('doc','slot')")
|
||||
self.db.execute("INSERT INTO source_revision VALUES('revision','doc',%s,%s,'legacy',%s)",(str(archive),br.sha(archive),archive.stat().st_size))
|
||||
self.db.execute("INSERT INTO source_candidate VALUES('imported','IMPORTED','slot',%s,NULL,%s,'legacy',NULL)",(str(path),br.sha(path)))
|
||||
self.assertEqual(ret.cleanup_drafts(self.db,'current',{},time.time(),br)[0],1)
|
||||
self.assertFalse(path.exists());self.assertTrue(archive.exists())
|
||||
self.assertEqual(self.db.rows('SELECT source_path FROM source_candidate')[0]['source_path'],str(archive))
|
||||
|
||||
def test_outside_file_is_never_deleted(self):
|
||||
path=self.add_draft(where=self.root/'unmanaged/private.xlsx')
|
||||
ret.cleanup_drafts(self.db,'current',{},time.time(),br)
|
||||
self.assertTrue(path.exists())
|
||||
|
||||
def test_symlink_is_not_followed_or_deleted(self):
|
||||
outside=self.root/'important.xlsx';outside.write_bytes(b'keep this')
|
||||
link=self.root/'runtimedata/archive/upload-drafts/upload-link.xlsx';link.parent.mkdir(parents=True);link.symlink_to(outside)
|
||||
self.db.execute('INSERT INTO source_upload_draft VALUES(%s,%s,%s,%s,%s,NULL,TIMESTAMPADD(HOUR,-30,CURRENT_TIMESTAMP),TIMESTAMPADD(HOUR,-25,CURRENT_TIMESTAMP))',('draft','link.xlsx',str(link),br.sha(outside),outside.stat().st_size))
|
||||
ret.cleanup_drafts(self.db,'current',{},time.time(),br)
|
||||
self.assertTrue(outside.exists());self.assertTrue(link.is_symlink())
|
||||
|
||||
def test_file_changed_after_checksum_is_not_deleted(self):
|
||||
path=self.add_draft();original=ret.fingerprint;calls=0
|
||||
def changed(p):
|
||||
nonlocal calls
|
||||
calls+=1
|
||||
if calls==2:path.write_bytes(b'new file contents')
|
||||
return original(p)
|
||||
with patch.object(ret,'fingerprint',side_effect=changed):
|
||||
count,size=ret.cleanup_drafts(self.db,'current',{},time.time(),br)
|
||||
self.assertEqual((count,size),(0,0));self.assertTrue(path.exists())
|
||||
self.assertTrue(self.db.rows('SELECT * FROM source_upload_draft'))
|
||||
|
||||
def test_new_reference_before_file_unlink_is_rechecked(self):
|
||||
path=self.add_draft();freeze=self.db.frozen;calls=0
|
||||
@contextmanager
|
||||
def insert_reference(expected):
|
||||
nonlocal calls
|
||||
calls+=1
|
||||
with freeze(expected):
|
||||
if calls==2:self.db.execute("INSERT INTO source_candidate VALUES('new','DISCOVERED','slot',%s,NULL,%s,'legacy',NULL)",(str(path),br.sha(path)))
|
||||
yield
|
||||
with patch.object(self.db,'frozen',side_effect=insert_reference):
|
||||
ret.cleanup_drafts(self.db,'current',{},time.time(),br)
|
||||
self.assertTrue(path.exists());self.assertFalse(self.db.rows('SELECT * FROM source_upload_draft'))
|
||||
|
||||
def test_long_running_sql_is_not_interrupted_for_cleanup(self):
|
||||
other=ret.Database(ROOT,self.name)
|
||||
try:
|
||||
other.conn.autocommit(False);other.rows('SELECT * FROM fact_black WHERE id=1 FOR UPDATE')
|
||||
with self.assertRaises(ret.Busy):
|
||||
with self.db.frozen('current'):pass
|
||||
self.assertTrue(other.rows('SELECT 1'));other.conn.rollback()
|
||||
finally:other.conn.close()
|
||||
|
||||
|
||||
def datetime_from_future():
|
||||
from datetime import datetime,timedelta,timezone
|
||||
return datetime.now(timezone.utc).replace(tzinfo=None)+timedelta(hours=1)
|
||||
|
||||
|
||||
class PrivateVerifierTests(unittest.TestCase):
|
||||
def test_small_backup_restores_on_private_server_and_cleans_up(self):
|
||||
import backup
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root=Path(temporary).resolve();(root/'.runtime').mkdir()
|
||||
(root/'.runtime/mysql').symlink_to(ROOT/'.runtime/mysql',target_is_directory=True)
|
||||
base=root/'output/backups';folder=base/'20261109T020000Z-12345678';folder.mkdir(parents=True)
|
||||
source=root/'source';source.mkdir();(source/'config.json').write_text('{}')
|
||||
with tarfile.open(folder/'application.tar.gz','w:gz') as tar:tar.add(source/'config.json',arcname='config.json')
|
||||
with tarfile.open(folder/'archive.tar.gz','w:gz'):pass
|
||||
sql="CREATE TABLE current_release(dataset_id varchar(20),release_id varchar(36)); INSERT INTO current_release VALUES('quality',NULL); CREATE TABLE source_revision(id varchar(36),archive_path varchar(1024),sha256 char(64),file_size bigint);"
|
||||
with gzip.open(folder/'database.sql.gz','wt') as out:out.write(sql)
|
||||
manifest={'formatVersion':3,'createdAt':'2026-11-09T02:00:00+00:00','files':{},'archiveSourceRoot':str(root/'runtimedata/archive'),
|
||||
'applicationFiles':{'config.json':br.sha(source/'config.json')},'requiredApplicationFiles':['config.json'],
|
||||
'tableCounts':{'current_release':1,'source_revision':0},'queryEvidence':{'releaseId':None,'months':[]}}
|
||||
for name,key in br.ARTIFACTS.items():manifest[key]=br.sha(folder/name)
|
||||
br.atomic(folder/'manifest.json',manifest)
|
||||
with patch.object(backup,'ROOT',root),patch.object(backup,'BACKUP_ROOT',base),patch.object(backup,'REQUIRED_APPLICATION_FILES',('config.json',)):
|
||||
result=backup.verify(folder)
|
||||
self.assertEqual(result['status'],'ISOLATED_RESTORE_VERIFIED')
|
||||
self.assertFalse(list((root/'.runtime/retention').glob('verify-*')))
|
||||
self.assertFalse(list((root/'.runtime/retention/scratch').glob('*.json')))
|
||||
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
@@ -0,0 +1,237 @@
|
||||
"""Exercise source retirement against an owned disposable MySQL schema and files."""
|
||||
from contextlib import contextmanager
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import uuid
|
||||
from unittest.mock import patch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / 'scripts'))
|
||||
import backup_retention as br
|
||||
import revision_retention as revisions
|
||||
spec = importlib.util.spec_from_file_location('source_retention_host', Path(__file__).with_name('retention.py'))
|
||||
ret = importlib.util.module_from_spec(spec); spec.loader.exec_module(ret)
|
||||
|
||||
|
||||
class SourceRevisionTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
if not (ROOT / '.runtime/local-config.json').is_file(): raise unittest.SkipTest('Local MySQL test runtime is not configured')
|
||||
cfg = json.loads((ROOT / '.runtime/local-config.json').read_text())
|
||||
cls.admin = ret.pymysql.connect(unix_socket=str(ROOT / '.runtime/mysql.sock'), user='root',
|
||||
password=cfg['rootPassword'], autocommit=True)
|
||||
cls.name = 'quality_retention_test_' + uuid.uuid4().hex[:12]
|
||||
ddl = [
|
||||
'CREATE TABLE data_release(id varchar(36) PRIMARY KEY,status varchar(24))',
|
||||
'CREATE TABLE current_release(dataset_id varchar(20) PRIMARY KEY,release_id varchar(36))',
|
||||
'CREATE TABLE source_document(id varchar(36) PRIMARY KEY,source_slot varchar(100))',
|
||||
"CREATE TABLE source_revision(id varchar(36) PRIMARY KEY,source_document_id varchar(36),archive_path varchar(1000),sha256 char(64),file_size bigint,status varchar(24) DEFAULT 'TRANSFORMED',replaces_revision_id varchar(36),parser_config json)",
|
||||
'CREATE TABLE transform_build(id varchar(36) PRIMARY KEY,source_revision_id varchar(36),FOREIGN KEY(source_revision_id) REFERENCES source_revision(id))',
|
||||
'CREATE TABLE release_member(release_id varchar(36),source_revision_id varchar(36),FOREIGN KEY(source_revision_id) REFERENCES source_revision(id))',
|
||||
'CREATE TABLE fact_black(id int PRIMARY KEY,source_revision_id varchar(36),raw_values json,components json,FOREIGN KEY(source_revision_id) REFERENCES source_revision(id))',
|
||||
'CREATE TABLE source_candidate(id varchar(36) PRIMARY KEY,status varchar(24),sha256 char(64),source_path varchar(1000),archive_path varchar(1000),replacement_revision_id varchar(36))',
|
||||
'CREATE TABLE source_upload_draft(id varchar(36) PRIMARY KEY,source_path varchar(1000))',
|
||||
'CREATE TABLE task(id int PRIMARY KEY,status varchar(24),payload json)',
|
||||
'CREATE TABLE source_change_request(id int PRIMARY KEY,original_revision_id varchar(36),status varchar(24))',
|
||||
'CREATE TABLE saved_context(id int PRIMARY KEY,context_json json,attachment_path varchar(1000))',
|
||||
'CREATE TABLE unknown_reference(id int PRIMARY KEY,revision_ref varchar(36),FOREIGN KEY(revision_ref) REFERENCES source_revision(id))']
|
||||
try:
|
||||
with cls.admin.cursor() as c:
|
||||
c.execute('CREATE DATABASE ' + ret.ident(cls.name)); c.execute('USE ' + ret.ident(cls.name))
|
||||
for statement in ddl:
|
||||
c.execute(statement + ' ENGINE=InnoDB')
|
||||
except BaseException:
|
||||
with cls.admin.cursor() as c: c.execute('DROP DATABASE IF EXISTS ' + ret.ident(cls.name))
|
||||
cls.admin.close(); raise
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
with cls.admin.cursor() as c: c.execute('DROP DATABASE ' + ret.ident(cls.name))
|
||||
cls.admin.close()
|
||||
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory(); self.root = Path(self.temp.name).resolve()
|
||||
self.db = ret.Database(ROOT, self.name); self.db.root = self.root
|
||||
self.db.execute('SET FOREIGN_KEY_CHECKS=0')
|
||||
for table in self.db.tables: self.db.execute('TRUNCATE TABLE ' + ret.ident(table))
|
||||
self.db.execute('SET FOREIGN_KEY_CHECKS=1')
|
||||
self.db.execute("INSERT INTO source_document VALUES('doc','black:2026:01')")
|
||||
self.db.execute("INSERT INTO data_release VALUES('current','PUBLISHED')")
|
||||
self.db.execute("INSERT INTO current_release VALUES('quality','current')")
|
||||
self.current_file = self.add_revision('current-rev', b'current workbook')
|
||||
self.db.execute("INSERT INTO release_member VALUES('current','current-rev')")
|
||||
self.db.execute("INSERT INTO transform_build VALUES('current-build','current-rev')")
|
||||
self.cleaner = revisions.Cleaner(self.db, br, ret.safe_file, ret.fingerprint, ret.Busy)
|
||||
self.state = {}
|
||||
|
||||
def tearDown(self):
|
||||
self.db.conn.close(); self.temp.cleanup()
|
||||
|
||||
def add_revision(self, rid='old-rev-unique', content=b'old workbook'):
|
||||
digest = __import__('hashlib').sha256(content).hexdigest()
|
||||
path = self.root / 'runtimedata/archive/source-revisions/black/2026/01' / (digest + '.xlsx')
|
||||
path.parent.mkdir(parents=True, exist_ok=True); path.write_bytes(content)
|
||||
self.db.execute('INSERT INTO source_revision(id,source_document_id,archive_path,sha256,file_size) VALUES(%s,%s,%s,%s,%s)',
|
||||
(rid, 'doc', str(path), digest, len(content)))
|
||||
return path
|
||||
|
||||
def collect(self, now):
|
||||
return self.cleaner.collect('current', self.state, now)
|
||||
|
||||
def present(self, rid='old-rev-unique'):
|
||||
return bool(self.db.rows('SELECT id FROM source_revision WHERE id=%s', (rid,)))
|
||||
|
||||
def test_unreferenced_revision_waits_grace_then_removes_only_its_owned_file(self):
|
||||
path = self.add_revision()
|
||||
self.assertEqual(self.collect(100)['sourceRevisionsRemoved'], 0)
|
||||
result = self.collect(2000)
|
||||
self.assertEqual(result['sourceRevisionsRemoved'], 1)
|
||||
self.assertEqual(result['sourceArchiveFilesRemoved'], 1)
|
||||
self.assertEqual(result['sourceArchiveBytesRemoved'], len(b'old workbook'))
|
||||
self.assertFalse(self.present()); self.assertFalse(path.exists())
|
||||
self.assertTrue(self.present('current-rev')); self.assertTrue(self.current_file.exists())
|
||||
self.assertTrue(self.db.rows('SELECT * FROM source_document'))
|
||||
|
||||
def test_any_release_build_fact_or_unknown_foreign_key_protects_revision(self):
|
||||
for rid, statement in [
|
||||
('candidate', "INSERT INTO release_member VALUES('candidate','candidate')"),
|
||||
('build', "INSERT INTO transform_build VALUES('old-build','build')"),
|
||||
('fact', "INSERT INTO fact_black VALUES(1,'fact',JSON_OBJECT(),JSON_OBJECT())"),
|
||||
('future', "INSERT INTO unknown_reference VALUES(1,'future')")]:
|
||||
self.add_revision(rid, rid.encode()); self.db.execute(statement)
|
||||
self.collect(100); self.assertEqual(self.collect(2000)['sourceRevisionsRemoved'], 0)
|
||||
for rid in ('candidate', 'build', 'fact', 'future'): self.assertTrue(self.present(rid))
|
||||
|
||||
def test_replacement_withdrawal_and_saved_json_ids_without_fks_are_protected(self):
|
||||
for rid in ('replacement', 'withdrawal', 'lineage', 'saved'): self.add_revision(rid, rid.encode())
|
||||
self.db.execute("INSERT INTO source_candidate VALUES('candidate','DISCOVERED',NULL,NULL,NULL,'replacement')")
|
||||
self.db.execute("INSERT INTO source_change_request VALUES(1,'withdrawal','APPROVED')")
|
||||
self.db.execute("UPDATE source_revision SET replaces_revision_id='lineage' WHERE id='current-rev'")
|
||||
self.db.execute("INSERT INTO saved_context VALUES(1,JSON_OBJECT('sourceRevisionId','saved'),NULL)")
|
||||
self.collect(100); self.assertEqual(self.collect(2000)['sourceRevisionsRemoved'], 0)
|
||||
|
||||
def test_catalog_hash_protects_original_even_with_external_path(self):
|
||||
path = self.add_revision(); digest = br.sha(path)
|
||||
self.db.execute("INSERT INTO source_candidate VALUES('catalog','IMPORTED',%s,'/external/original.xlsx',NULL,NULL)", (digest,))
|
||||
self.collect(100); self.assertEqual(self.collect(2000)['sourceRevisionsRemoved'], 0)
|
||||
self.assertTrue(self.present()); self.assertTrue(path.exists())
|
||||
|
||||
def test_new_reference_resets_continuous_grace(self):
|
||||
self.add_revision(); self.collect(100)
|
||||
self.db.execute("INSERT INTO unknown_reference VALUES(1,'old-rev-unique')"); self.collect(1500)
|
||||
self.db.execute('DELETE FROM unknown_reference')
|
||||
self.assertEqual(self.collect(2000)['sourceRevisionsRemoved'], 0)
|
||||
self.assertEqual(self.collect(3801)['sourceRevisionsRemoved'], 1)
|
||||
|
||||
def test_revision_metadata_change_restarts_grace(self):
|
||||
self.add_revision(); self.collect(100)
|
||||
self.db.execute("UPDATE source_revision SET parser_config=JSON_OBJECT('changed',TRUE) WHERE id='old-rev-unique'")
|
||||
self.assertEqual(self.collect(2000)['sourceRevisionsRemoved'], 0)
|
||||
self.assertEqual(self.collect(3801)['sourceRevisionsRemoved'], 1)
|
||||
|
||||
def test_shared_archive_stays_until_its_other_revision_is_unreferenced(self):
|
||||
path = self.add_revision(content=b'current workbook')
|
||||
self.collect(100); result = self.collect(2000)
|
||||
self.assertEqual(result['sourceRevisionsRemoved'], 1)
|
||||
self.assertEqual(result['sourceArchiveFilesRemoved'], 0)
|
||||
self.assertTrue(path.exists()); self.assertTrue(self.present('current-rev'))
|
||||
|
||||
def test_saved_file_path_protects_file_after_metadata_removal(self):
|
||||
path = self.add_revision()
|
||||
self.db.execute('INSERT INTO saved_context VALUES(1,JSON_OBJECT(\'archivePath\',%s),NULL)', (str(path),))
|
||||
self.collect(100); result = self.collect(2000)
|
||||
self.assertEqual(result['sourceRevisionsRemoved'], 1)
|
||||
self.assertEqual(result['sourceFilesPending'], 1); self.assertTrue(path.exists())
|
||||
self.db.execute('DELETE FROM saved_context')
|
||||
self.assertEqual(self.cleaner.drain('current')[0], 1); self.assertFalse(path.exists())
|
||||
|
||||
def test_external_symlink_mismatched_or_incomplete_files_are_kept(self):
|
||||
for rid in ('outside', 'symlink', 'damaged', 'incomplete'): self.add_revision(rid, rid.encode())
|
||||
external = self.root / 'important.xlsx'; external.write_bytes(b'outside')
|
||||
self.db.execute("UPDATE source_revision SET archive_path=%s WHERE id='outside'", (str(external),))
|
||||
row = self.db.rows("SELECT archive_path FROM source_revision WHERE id='symlink'")[0]
|
||||
link = Path(row['archive_path']); link.unlink(); link.symlink_to(external)
|
||||
row = self.db.rows("SELECT archive_path FROM source_revision WHERE id='damaged'")[0]
|
||||
Path(row['archive_path']).write_bytes(b'changed')
|
||||
self.db.execute("UPDATE source_revision SET status='RECEIVED' WHERE id='incomplete'")
|
||||
self.collect(100); self.assertEqual(self.collect(2000)['sourceRevisionsRemoved'], 0)
|
||||
for rid in ('outside', 'symlink', 'damaged', 'incomplete'): self.assertTrue(self.present(rid))
|
||||
self.assertTrue(external.exists()); self.assertTrue(link.is_symlink())
|
||||
|
||||
def test_reference_added_before_database_delete_is_rechecked(self):
|
||||
path = self.add_revision(); self.collect(100); original = self.db.frozen
|
||||
@contextmanager
|
||||
def add_reference(current):
|
||||
with original(current):
|
||||
self.db.execute("INSERT INTO unknown_reference VALUES(1,'old-rev-unique')")
|
||||
yield
|
||||
with patch.object(self.db, 'frozen', side_effect=add_reference): result = self.collect(2000)
|
||||
self.assertEqual(result['sourceRevisionsRemoved'], 0)
|
||||
self.assertTrue(path.exists()); self.assertTrue(self.present())
|
||||
|
||||
def test_file_changed_after_hash_is_kept(self):
|
||||
path = self.add_revision(); self.collect(100); original = self.db.frozen
|
||||
@contextmanager
|
||||
def change_file(current):
|
||||
path.write_bytes(b'file edited during preflight')
|
||||
with original(current): yield
|
||||
with patch.object(self.db, 'frozen', side_effect=change_file): result = self.collect(2000)
|
||||
self.assertEqual(result['sourceRevisionsRemoved'], 0)
|
||||
self.assertTrue(path.exists()); self.assertTrue(self.present())
|
||||
|
||||
def test_crash_after_database_commit_resumes_registered_file_removal(self):
|
||||
path = self.add_revision(); self.collect(100)
|
||||
with patch.object(self.cleaner, 'drain', side_effect=RuntimeError('interrupted')):
|
||||
with self.assertRaises(RuntimeError): self.collect(2000)
|
||||
self.assertFalse(self.present()); self.assertTrue(path.exists())
|
||||
self.assertEqual(self.cleaner.drain('current')[0], 1)
|
||||
self.assertFalse(path.exists()); self.assertTrue(self.current_file.exists())
|
||||
|
||||
def test_database_rollback_keeps_file_and_discards_only_journal(self):
|
||||
path = self.add_revision(); self.collect(100); original = self.db.execute
|
||||
def rollback(sql, args=()):
|
||||
result = original(sql, args)
|
||||
if sql.startswith('DELETE FROM source_revision'): raise RuntimeError('rollback')
|
||||
return result
|
||||
with patch.object(self.db, 'execute', side_effect=rollback):
|
||||
with self.assertRaises(RuntimeError): self.collect(2000)
|
||||
self.assertTrue(self.present()); self.assertTrue(path.exists())
|
||||
self.assertEqual(self.cleaner.drain('current'), (0, 0))
|
||||
self.assertFalse(list(self.cleaner.journals.glob('*.json')))
|
||||
|
||||
def test_new_path_reference_before_unlink_protects_file(self):
|
||||
path = self.add_revision(); self.collect(100); original = self.cleaner.drain
|
||||
def add_reference(current, limit=5):
|
||||
self.db.execute('INSERT INTO source_upload_draft VALUES(\'new\',%s)', (str(path),))
|
||||
return original(current, limit)
|
||||
with patch.object(self.cleaner, 'drain', side_effect=add_reference): result = self.collect(2000)
|
||||
self.assertEqual(result['sourceRevisionsRemoved'], 1)
|
||||
self.assertEqual(result['sourceArchiveFilesRemoved'], 0); self.assertTrue(path.exists())
|
||||
|
||||
def test_running_task_or_changed_publication_defers_without_deleting(self):
|
||||
path = self.add_revision(); self.collect(100)
|
||||
self.db.execute("INSERT INTO task VALUES(1,'RUNNING',NULL)")
|
||||
with self.assertRaises(ret.Busy): self.collect(2000)
|
||||
self.assertTrue(self.present()); self.assertTrue(path.exists())
|
||||
self.db.execute('DELETE FROM task')
|
||||
self.db.execute("UPDATE current_release SET release_id='different'")
|
||||
with self.assertRaises(ret.Busy): self.collect(2000)
|
||||
self.assertTrue(self.present()); self.assertTrue(path.exists())
|
||||
|
||||
def test_unsafe_registry_and_unknown_stray_files_are_not_deleted(self):
|
||||
path = self.add_revision(); self.collect(100)
|
||||
self.cleaner.journals.mkdir(parents=True)
|
||||
br.atomic(self.cleaner.journals / ('1' * 32 + '.json'), {'kind': 'quality-source-retirement'})
|
||||
unknown = path.with_name('unregistered.xlsx'); unknown.write_bytes(b'keep')
|
||||
self.cleaner.drain('current'); self.assertTrue(unknown.exists()); self.assertTrue(path.exists())
|
||||
with self.cleaner.quick_checks():
|
||||
self.cleaner.reference_deadline = time.monotonic() - 1
|
||||
with self.assertRaises(ret.Busy): self.cleaner.exists('source_revision', 'id=%s', ('old-rev-unique',))
|
||||
|
||||
|
||||
if __name__ == '__main__': unittest.main()
|
||||
@@ -0,0 +1,116 @@
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
|
||||
spec = importlib.util.spec_from_file_location('quality_macos_service', Path(__file__).with_name('service.py'))
|
||||
service = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(service)
|
||||
|
||||
|
||||
class Fixture(service.Guardian):
|
||||
def command(self, role):
|
||||
return ['/bin/sleep', '90' if role == 'mysql' else '91']
|
||||
|
||||
def occupied(self, role):
|
||||
return False
|
||||
|
||||
def environment(self):
|
||||
return os.environ.copy()
|
||||
|
||||
def db_ready(self):
|
||||
return self.ready
|
||||
|
||||
def backend_ready(self):
|
||||
return service.matches(self.records.get('backend')) is True
|
||||
|
||||
|
||||
class GuardianTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.temp.name)
|
||||
(self.root / '.runtime').mkdir()
|
||||
self.g = Fixture({'root': str(self.root)}, self.root / 'guardian')
|
||||
self.g.value = {'mysqlPort': 59998}
|
||||
self.g.ready = False
|
||||
self.external = []
|
||||
|
||||
def tearDown(self):
|
||||
for p in list(self.g.children.values()) + self.external:
|
||||
if p.poll() is None:
|
||||
p.terminate()
|
||||
p.wait(timeout=10)
|
||||
self.temp.cleanup()
|
||||
|
||||
def test_database_must_be_ready_before_backend_starts(self):
|
||||
self.g.tick()
|
||||
self.assertIn('mysql', self.g.records)
|
||||
self.assertNotIn('backend', self.g.records)
|
||||
self.g.ready = True
|
||||
self.g.tick()
|
||||
self.assertTrue(service.matches(self.g.records['backend']))
|
||||
|
||||
def test_abnormal_child_exit_is_restarted_without_duplicate_mysql(self):
|
||||
self.g.ready = True
|
||||
self.g.tick()
|
||||
mysql = self.g.records['mysql']['pid']
|
||||
old = self.g.children['backend']
|
||||
old.kill()
|
||||
old.wait()
|
||||
self.g.tick()
|
||||
self.assertNotIn('backend', self.g.records)
|
||||
self.assertEqual(self.g.records['mysql']['pid'], mysql)
|
||||
self.g.next_start['backend'] = 0
|
||||
self.g.tick()
|
||||
self.assertNotEqual(self.g.records['backend']['pid'], old.pid)
|
||||
self.assertEqual(self.g.restarts, {'mysql': 1, 'backend': 2})
|
||||
|
||||
def test_database_exit_is_restarted_before_missing_backend(self):
|
||||
self.g.tick()
|
||||
old = self.g.children['mysql']
|
||||
old.kill()
|
||||
old.wait()
|
||||
self.g.tick()
|
||||
self.assertNotIn('mysql', self.g.records)
|
||||
self.g.next_start['mysql'] = 0
|
||||
self.g.tick()
|
||||
self.assertNotEqual(self.g.records['mysql']['pid'], old.pid)
|
||||
self.assertNotIn('backend', self.g.records)
|
||||
|
||||
def test_existing_process_is_adopted_and_unknown_process_is_not_replaced(self):
|
||||
command = self.g.command('mysql')
|
||||
p = subprocess.Popen(command, start_new_session=True)
|
||||
self.external.append(p)
|
||||
info = service.record(p.pid, command)
|
||||
service.atomic(self.root / '.runtime/mysql.process.json', {'pid': p.pid, 'started': info['started']})
|
||||
self.assertTrue(self.g.ensure('mysql'))
|
||||
self.assertEqual(self.g.restarts['mysql'], 0)
|
||||
self.assertIsNone(service.matches(info, self.g.command('backend')))
|
||||
self.assertIs(p.poll(), None)
|
||||
|
||||
def test_missing_data_marker_never_initializes_database(self):
|
||||
service.atomic(self.root / '.runtime/local-config.json', {'databaseReady': True})
|
||||
real = service.Guardian({'root': str(self.root), 'jarSha256': '0'*64}, self.root / 'guardian')
|
||||
with self.assertRaises(RuntimeError):
|
||||
real.preflight()
|
||||
self.assertFalse((self.root / '.runtime/mysql-data').exists())
|
||||
|
||||
def test_pinned_jar_mismatch_is_rejected(self):
|
||||
r = self.root / '.runtime'
|
||||
(r / 'mysql-data').mkdir()
|
||||
(r / 'mysql-data/auto.cnf').write_text('existing marker')
|
||||
(r / 'backend-builds').mkdir()
|
||||
(r / 'backend-builds' / ('0'*64+'.jar')).write_bytes(b'unaccepted')
|
||||
service.atomic(r / 'local-config.json', {'databaseReady': True})
|
||||
real = service.Guardian({'root': str(self.root), 'jarSha256': '0'*64}, self.root / 'guardian')
|
||||
with self.assertRaises(RuntimeError):
|
||||
real.preflight()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user