新增:Linux Docker 部署与自动回收、备份及容量监测
This commit is contained in:
@@ -0,0 +1,12 @@
|
||||
# 此文件只放非敏感选项,真实密码由 manage.py init 生成。
|
||||
HTTP_BIND=127.0.0.1
|
||||
HTTP_PORT=5280
|
||||
QUALITY_IMAGE_TAG=local
|
||||
MYSQL_DATABASE=quality_platform
|
||||
MYSQL_USER=quality_app
|
||||
QUALITY_ADMIN_USERNAME=admin
|
||||
JAVA_MAX_HEAP=2048m
|
||||
# 从旧库迁移时必须核实旧版本会话的时区,不要猜测。
|
||||
MYSQL_LEGACY_TIMESTAMP_OFFSET_SECONDS=UNCONFIRMED
|
||||
QUALITY_ANALYSIS_PROTOCOL_V1_ENABLED=false
|
||||
QUALITY_ANALYSIS_AUTONOMOUS_ENABLED=false
|
||||
@@ -0,0 +1,52 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
FROM maven:3.9-eclipse-temurin-21 AS java-build
|
||||
WORKDIR /build
|
||||
COPY backend/pom.xml backend/pom.xml
|
||||
RUN --mount=type=cache,target=/root/.m2 mvn -B -f backend/pom.xml dependency:go-offline
|
||||
COPY backend/src backend/src
|
||||
# 默认执行不需要私有工作簿的测试;不把生产库传入构建。
|
||||
RUN --mount=type=cache,target=/root/.m2 mvn -B -f backend/pom.xml package
|
||||
|
||||
FROM python:3.11-slim-bookworm AS runtimes
|
||||
ARG TARGETARCH
|
||||
RUN test "$TARGETARCH" = amd64
|
||||
WORKDIR /build
|
||||
COPY deploy/docker/assets.lock.json deploy/docker/fetch_runtimes.py ./
|
||||
RUN python fetch_runtimes.py
|
||||
|
||||
FROM debian:bookworm-slim AS app
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
ca-certificates tzdata tini procps libaio1 libnuma1 libncurses6 libtirpc3 libstdc++6 libgomp1 \
|
||||
libatomic1 libcrypt1 libzstd1 liblz4-1 libx11-6 libxext6 libxi6 libxrender1 libxtst6 libasound2 fontconfig fonts-dejavu-core \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& groupadd -g 10001 quality && useradd -u 10001 -g quality -m quality
|
||||
COPY --from=runtimes /runtimes/jre /opt/java
|
||||
COPY --from=runtimes /runtimes/python /opt/python
|
||||
COPY --from=runtimes /runtimes/mysql /opt/mysql
|
||||
ENV JAVA_HOME=/opt/java PATH=/opt/nao/bin:/opt/java/bin:/opt/mysql/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
|
||||
PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 QUALITY_CONTAINER=1 TZ=Asia/Shanghai
|
||||
COPY deploy/docker/requirements-linux.lock deploy/macos/retention-requirements.txt /tmp/requirements/
|
||||
RUN /opt/python/bin/python3 -m venv /opt/nao \
|
||||
&& /opt/nao/bin/pip install --no-cache-dir --require-hashes -r /tmp/requirements/requirements-linux.lock -r /tmp/requirements/retention-requirements.txt \
|
||||
&& /opt/nao/bin/python -c "import nao_core, pymysql, psutil; import importlib.metadata as m; assert m.version('nao-core') == '0.3.26'" \
|
||||
&& /opt/java/bin/java -version && /opt/mysql/bin/mysqld --version \
|
||||
&& rm -rf /tmp/requirements
|
||||
WORKDIR /srv/quality
|
||||
COPY .dockerignore runtime.lock.json requirements-nao.in requirements-nao.lock ./
|
||||
COPY backend/pom.xml backend/pom.xml
|
||||
COPY backend/src backend/src
|
||||
COPY web/package.json web/pnpm-lock.yaml web/index.html web/tsconfig.json web/vite.config.ts web/pnpm-workspace.yaml web/
|
||||
COPY web/src web/src
|
||||
COPY web/public web/public
|
||||
COPY scripts scripts
|
||||
COPY ai ai
|
||||
COPY deploy/macos deploy/macos
|
||||
COPY deploy/docker deploy/docker
|
||||
COPY --from=java-build /build/backend/target/quality-platform-backend-0.1.0-SNAPSHOT.jar app/backend.jar
|
||||
RUN sha256sum app/backend.jar | cut -d ' ' -f 1 > app/backend.sha256 \
|
||||
&& mkdir -p .runtime output/backups output/logs runtimedata/archive \
|
||||
&& chown -R quality:quality .runtime output runtimedata
|
||||
USER 10001:10001
|
||||
EXPOSE 5281
|
||||
ENTRYPOINT ["/usr/bin/tini", "--", "/opt/nao/bin/python", "deploy/docker/app.py"]
|
||||
CMD ["serve"]
|
||||
@@ -0,0 +1,13 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
FROM node:22-bookworm-slim AS build
|
||||
RUN npm install --global pnpm@10.15.1
|
||||
WORKDIR /build
|
||||
COPY web/package.json web/pnpm-lock.yaml web/pnpm-workspace.yaml ./
|
||||
RUN --mount=type=cache,target=/root/.local/share/pnpm/store pnpm install --frozen-lockfile
|
||||
COPY web ./
|
||||
RUN pnpm build
|
||||
|
||||
FROM nginx:stable-alpine
|
||||
COPY deploy/docker/nginx.conf /etc/nginx/conf.d/default.conf
|
||||
COPY --from=build /build/dist /usr/share/nginx/html
|
||||
EXPOSE 8080
|
||||
@@ -0,0 +1,76 @@
|
||||
# Linux Docker 部署
|
||||
|
||||
本目录是项目源码的一部分,随 Gitea 仓库同步。它提供手动构建与运行入口,不需要 Gitea Actions、Runner 或自动发布流水线。构建需要联网获取公共依赖;已构建镜像在启动时不运行 Maven、pnpm、pip 或下载运行时。AI 调用仍需要可用的模型服务。
|
||||
|
||||
当前准备目标为 **Linux x86_64 / Docker Engine / Compose V2(建议 2.24 以上)**。应用运行时固定为 Java 21.0.13、Python 3.11.15、MySQL 8.4.13、nao-core 0.3.26。原生运行时校验和在 `assets.lock.json`,Python 依赖使用带哈希的锁文件。基础系统镜像按发行系列获取安全更新;正式发布时应记录构建提交和镜像 digest。ARM64 不在本版本支持范围内。
|
||||
|
||||
## 服务与数据
|
||||
|
||||
| 服务 | 内容 | 数据位置 |
|
||||
| --- | --- | --- |
|
||||
| mysql | MySQL,关闭 binary log,保留事务 redo/undo | `mysql_data` 命名卷 |
|
||||
| app | 后端、AI、自动回收、备份验证、容量监测 | `state`、`archives`、`backups`、`logs` 命名卷 |
|
||||
| web | 前端与 API 代理,支持流式响应 | 镜像内静态文件 |
|
||||
|
||||
只发布网页端口,默认 `127.0.0.1:5280`;MySQL 和后端没有宿主机端口映射。应用以 UID 10001 运行,镜像只读,不挂载 Docker socket。后端及所有维护子进程共享同一容器的进程空间,避免误判其他容器内的校验任务已经退出。不要同时启动第二个 app 副本访问同一组数据卷。
|
||||
|
||||
首次初始化后,密码存放于项目私有目录 `.runtime/docker/secrets/`,不是源码、镜像层或命令参数。Compose 从管理入口的环境变量读取密码,再作为 secret 文件交给容器。主机管理员仍能查看容器配置,需限制 Docker 管理权限。数据库初始化后不要直接重新生成密码;密码轮换必须同步修改数据库账户。
|
||||
|
||||
## 从仓库构建与启动
|
||||
|
||||
在项目根目录执行:
|
||||
|
||||
```sh
|
||||
python3 deploy/docker/manage.py init
|
||||
python3 deploy/docker/manage.py config
|
||||
python3 deploy/docker/manage.py build
|
||||
python3 deploy/docker/manage.py up
|
||||
python3 deploy/docker/manage.py status
|
||||
```
|
||||
|
||||
`init` 不覆盖已有设置或密码。非敏感设置位于 `.runtime/docker/config.env`;内网访问时可把 `HTTP_BIND` 改为服务器内网 IP,配合防火墙限制访问范围。管理员初始密码位于私有文件 `.runtime/docker/secrets/admin_password`,登录后及时更改。首次克隆为空系统,业务数据需单独授权导入。
|
||||
|
||||
启用服务器上的 Docker 开机启动后,已创建且未手动停止的容器通过 `restart: unless-stopped` 随 Docker 自动启动;MySQL 就绪后才启动应用,应用就绪后再启动网页。后端异常退出会使 app 退出并由 Docker 重启;维护进程退出由入口进程重新拉起。健康检查用于报告就绪状态,不会因一次查询超时强杀正在工作的数据库。
|
||||
|
||||
```sh
|
||||
sudo systemctl enable --now docker
|
||||
python3 deploy/docker/manage.py logs
|
||||
python3 deploy/docker/manage.py stop
|
||||
```
|
||||
|
||||
`stop` 保留所有数据卷,并表示主动停服;之后用 `up` 恢复。不要执行 `docker compose down -v`,它会删除数据库、归档和备份卷。操作系统关机时仍会停服;要摆脱测试 Mac 的关机和网络影响,必须将整套服务部署在持续开机联网的服务器上。
|
||||
|
||||
## 自动控制空间增长
|
||||
|
||||
复用 `deploy/macos/retention.py` 中的平台通用回收逻辑和 `scripts/` 内的保留模块;macOS 的 launchd 安装代码不会在 Linux 执行。
|
||||
|
||||
- 每 5 分钟检查旧版本、终态无引用构建、过期上传草稿和无引用旧来源修订。当前快照、候选基线、保存的报表及分析引用均保护;删除前重新确认当前版本与任务状态。
|
||||
- 导入、分析、排队、重试或未知状态任务存在时延期,锁冲突也延期,不终止任务。无引用对象连续 30 分钟后才进入回收;上传草稿先经过 24 小时有效期。
|
||||
- 发现新正式版本或距上次备份超过 24 小时,在空闲时生成候选备份。新候选通过独立、无网络 MySQL 实例恢复、表行数、业务汇总、来源归档及应用文件校验后,才替换上一份有效备份。失败时保留旧备份;创建期间会短暂同时存在新旧两份及恢复校验空间。
|
||||
- 测试数据库按所有者、进程身份、服务器 UUID 和标记登记。正常完成即清理;异常遗留超过 24 小时且所有者及子进程已退出后再处理。不按名称前缀批量删除未知数据库。新增数据库集成测试应使用 `OwnedTestDatabase` 或 `managed_test_resources.py`,并在 finally 中释放。
|
||||
- Java 日志按日期及每段 10MB 滚动,保留 7 天,归档总量上限 80MB,为活动文件留出空间。关闭的其他项目日志每类最多 100MiB、最长 7 天;活动文件受保护。Docker 每个容器日志使用 `local` 驱动、10m×10 段限制。MySQL 错误日志进入受限制的容器日志;binary log 关闭后不支持历史时间点恢复,恢复依赖最新有效快照。
|
||||
- 容量每 5 分钟采样,保留 30 天日报。业务数据、测试/校验临时空间分别达到 30GiB 提醒、50GiB 严重提醒;磁盘余量低于 50/20GiB、一天增长超过 2GiB 且超过 20%、连续维护失败也告警。告警写入容器日志及 `state` 卷的 `operations/alerts.jsonl`,同类 6 小时内去重。阈值在 `state` 卷的 `operations/policy.json`,不应为压低统计而删除业务数据。
|
||||
|
||||
Linux 的 MySQL 用量来自 `information_schema` 表和索引估算,不包含系统表空间、redo/undo 和 Docker 镜像缓存;宿主机还应使用磁盘监测与 `docker system df` 监测这些空间。容器内报告的可用空间取各业务卷的最小值;MySQL 卷所在磁盘也要由主机监控。此版本不提供 Linux 桌面弹窗或外部通知推送,容器日志可接入主机现有告警系统。
|
||||
|
||||
手动生成并验证一份备份:
|
||||
|
||||
```sh
|
||||
python3 deploy/docker/manage.py backup
|
||||
```
|
||||
|
||||
维护记录分别在 `state` 卷的 `retention/state.json`、`operations/monitor.json`。运行包或数据库结构改变时,回收会暂停;检查新版本兼容性后才执行 `python3 deploy/docker/manage.py review-retention`。此命令只更新兼容性标记,不取消任务保护或备份校验。
|
||||
|
||||
## 数据迁移与公网入口
|
||||
|
||||
当前测试 Mac 不会因本目录更新而切换运行包、迁移数据库或改变公网入口。源码中也不包含测试库、来源表格、现有备份和 API 密钥。
|
||||
|
||||
正式迁移采用逻辑备份及来源归档,不把 Mac 的 MySQL 数据目录直接复制到 Linux。先做 `backup.py create-and-verify`,保留原机;在新服务器的新数据库和独立目录恢复并重绑定归档绝对路径,核对用户、当前发布版本、月度汇总、归档摘要、登录和 AI,再切换域名。`backup.py restore --help` 提供新库/新目录的防覆盖恢复入口;它不是自动覆盖容器数据卷的导入命令。旧库的时间迁移参数必须从原部署核实。
|
||||
|
||||
长期入口建议为:域名 → 服务器 HTTPS 反向代理 → 本服务器 web → app → MySQL。公网只开放 443(按需开放 80 做跳转/证书),不再依赖测试 Mac 的 SSH 隧道。另一条路径是服务器应用连接托管 MySQL,另配对象存储及异地备份;该方案需要单独评估数据库权限、备份方式及来源路径,不能直接套用本地卷配置。
|
||||
|
||||
## 验收边界
|
||||
|
||||
本目录当前已准备源码与测试;没有 Linux Docker 主机时,不把静态检查或 Mac 上的测试称作容器启动成功。上主机后至少完成:构建三项服务、空库初始化、登录/导入/AI、后端与 MySQL 异常退出重启、主机重启、有效备份恢复、新旧数据对账、进行中任务保护、日志滚动及告警、HTTPS 与断开测试 Mac 后访问。容量规划还应为新旧备份和独立恢复校验预留余量。
|
||||
|
||||
参考:[Compose 启动顺序](https://docs.docker.com/compose/how-tos/startup-order/)、[Compose 服务与重启策略](https://docs.docker.com/reference/compose-file/services/)、[MySQL 官方镜像](https://hub.docker.com/_/mysql)、[Logback 日志保留](https://logback.qos.ch/manual/appenders-rolling.html)。
|
||||
@@ -0,0 +1,161 @@
|
||||
#!/usr/bin/env python3
|
||||
"""容器入口:固定运行包、私有配置、子进程守护和安全停止。"""
|
||||
import argparse
|
||||
import fcntl
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path[:0] = [str(ROOT / 'scripts'), str(ROOT / 'deploy/macos')]
|
||||
|
||||
|
||||
def secret(name):
|
||||
path = Path('/run/secrets') / name
|
||||
value = path.read_text().strip()
|
||||
if not re.fullmatch(r'[A-Za-z0-9_-]{24,128}', value):
|
||||
raise ValueError('Invalid secret: ' + name)
|
||||
return value
|
||||
|
||||
|
||||
def identifier(value):
|
||||
if not re.fullmatch(r'[A-Za-z][A-Za-z0-9_]{0,47}', value):
|
||||
raise ValueError('Invalid database identifier')
|
||||
return value
|
||||
|
||||
|
||||
def prepare(root=ROOT):
|
||||
import backup_retention as br
|
||||
root = Path(root)
|
||||
for relative in ('.runtime/retention', '.runtime/operations', '.runtime/nao-users',
|
||||
'.runtime/nao-trial', 'runtimedata/archive', 'output/backups', 'output/logs'):
|
||||
(root / relative).mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
for relative, destination in (('.runtime/mysql', '/opt/mysql'), ('.runtime/nao-trial/venv', '/opt/nao')):
|
||||
path = root / relative
|
||||
if path.is_symlink():
|
||||
if str(path.readlink()) != destination: raise ValueError('Unexpected runtime link')
|
||||
elif path.exists(): raise ValueError('Runtime path already occupied')
|
||||
else: path.symlink_to(destination, target_is_directory=True)
|
||||
value = {'mysqlHost': 'mysql', 'mysqlPort': 3306,
|
||||
'mysqlDatabase': identifier(os.environ.get('MYSQL_DATABASE', 'quality_platform')),
|
||||
'mysqlUser': identifier(os.environ.get('MYSQL_USER', 'quality_app')),
|
||||
'mysqlPassword': secret('mysql_password'), 'rootPassword': secret('mysql_root_password'),
|
||||
'adminUsername': os.environ.get('QUALITY_ADMIN_USERNAME', 'admin'),
|
||||
'adminPassword': secret('admin_password'), 'databaseReady': True, 'binaryLogEnabled': False,
|
||||
'legacyTimestampOffsetSeconds': os.environ.get('MYSQL_LEGACY_TIMESTAMP_OFFSET_SECONDS', 'UNCONFIRMED')}
|
||||
config = root / '.runtime/local-config.json'
|
||||
if config.exists():
|
||||
old = json.loads(config.read_text())
|
||||
# 更改持久数据库的密码需要专门轮换,不能在重启时默默写入不一致配置。
|
||||
for key in ('mysqlDatabase', 'mysqlUser', 'mysqlPassword', 'rootPassword'):
|
||||
if old.get(key) != value[key]: raise ValueError('Persisted database settings changed: ' + key)
|
||||
br.atomic(config, value)
|
||||
cnf = root / '.runtime/mysql-root.cnf'
|
||||
cnf.write_text('[client]\nuser=root\npassword=' + value['rootPassword'] + '\nhost=mysql\nport=3306\nprotocol=TCP\n')
|
||||
cnf.chmod(0o600)
|
||||
import operations
|
||||
policy = root / '.runtime/operations/policy.json'
|
||||
if not policy.exists(): br.atomic(policy, operations.DEFAULTS)
|
||||
return value
|
||||
|
||||
|
||||
def backend_environment(value):
|
||||
env = os.environ.copy()
|
||||
env.update({'MYSQL_URL': 'jdbc:mysql://mysql:3306/' + value['mysqlDatabase'] +
|
||||
'?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC&forceConnectionTimeZoneToSession=true&useSSL=false&allowPublicKeyRetrieval=true',
|
||||
'MYSQL_USER': value['mysqlUser'], 'MYSQL_PASSWORD': value['mysqlPassword'],
|
||||
'QUALITY_BOOTSTRAP_ADMIN_USERNAME': value['adminUsername'],
|
||||
'QUALITY_BOOTSTRAP_ADMIN_PASSWORD': value['adminPassword'],
|
||||
'SERVER_ADDRESS': '0.0.0.0', 'QUALITY_PROJECT_ROOT': str(ROOT),
|
||||
'QUALITY_ARCHIVE_ROOT': str(ROOT / 'runtimedata/archive'), 'QUALITY_ALLOWED_ROOTS': '',
|
||||
'LOGGING_CONFIG': str(ROOT / 'deploy/macos/logback-quality.xml'),
|
||||
'QUALITY_LOG_DIR': str(ROOT / 'output/logs'), 'QUALITY_CONTAINER': '1'})
|
||||
return env
|
||||
|
||||
|
||||
def ready():
|
||||
try:
|
||||
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||
with opener.open('http://127.0.0.1:5281/actuator/health', timeout=3) as response:
|
||||
return json.load(response).get('status') == 'UP'
|
||||
except (OSError, ValueError): return False
|
||||
|
||||
|
||||
def review_retention(first=False):
|
||||
import backup_retention as br
|
||||
from retention import Database
|
||||
path = ROOT / '.runtime/retention/config.json'
|
||||
if first and path.exists(): return
|
||||
with br.lock(ROOT, timeout=0):
|
||||
db = Database(ROOT)
|
||||
try:
|
||||
if db.busy(): raise RuntimeError('Application jobs are active; review deferred')
|
||||
accepted = br.accepted_backend(ROOT)
|
||||
if not accepted: raise RuntimeError('Pinned backend not found')
|
||||
br.atomic(path, {'schemaSignature': db.schema_signature, 'acceptedJarSha256': accepted[1],
|
||||
'retirementGraceSeconds': 1800, 'intervalSeconds': 300})
|
||||
finally: db.conn.close()
|
||||
|
||||
|
||||
def serve():
|
||||
import backup_retention as br
|
||||
(ROOT / '.runtime').mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
guard = (ROOT / '.runtime/container.lock').open('a')
|
||||
fcntl.flock(guard, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
value = prepare()
|
||||
br.accepted_backend(ROOT)
|
||||
heap = os.environ.get('JAVA_MAX_HEAP', '2048m')
|
||||
if not re.fullmatch(r'[1-9][0-9]*[mMgG]', heap): raise ValueError('Invalid Java heap size')
|
||||
env = backend_environment(value)
|
||||
children = {'backend': subprocess.Popen(['java', '-Xms128m', '-Xmx' + heap, '-jar', str(ROOT / 'app/backend.jar')],
|
||||
env=env, start_new_session=True)}
|
||||
stopping = False
|
||||
def stop(signum, frame):
|
||||
nonlocal stopping
|
||||
stopping = True
|
||||
signal.signal(signal.SIGTERM, stop); signal.signal(signal.SIGINT, stop)
|
||||
try:
|
||||
while not stopping:
|
||||
if children['backend'].poll() is not None: return 1
|
||||
if ready():
|
||||
try: review_retention(first=True)
|
||||
except Exception as error: print('retention-initialization: ' + type(error).__name__, flush=True)
|
||||
for role in ('retention', 'monitor'):
|
||||
if role not in children or children[role].poll() is not None:
|
||||
children[role] = subprocess.Popen([sys.executable, str(ROOT / 'deploy/docker/worker.py'), role],
|
||||
env=env, start_new_session=True)
|
||||
time.sleep(1)
|
||||
return 0
|
||||
finally:
|
||||
for process in children.values():
|
||||
if process.poll() is None:
|
||||
try: os.killpg(process.pid, signal.SIGTERM)
|
||||
except ProcessLookupError: pass
|
||||
deadline = time.monotonic() + 270
|
||||
for process in children.values():
|
||||
try: process.wait(timeout=max(.1, deadline - time.monotonic()))
|
||||
except subprocess.TimeoutExpired:
|
||||
os.killpg(process.pid, signal.SIGKILL); process.wait()
|
||||
guard.close()
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action', choices=['serve', 'health', 'review-retention'], default='serve', nargs='?')
|
||||
args = parser.parse_args()
|
||||
if args.action == 'health': return 0 if ready() else 1
|
||||
if args.action == 'review-retention': review_retention(); return 0
|
||||
return serve()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try: raise SystemExit(main())
|
||||
except Exception as error:
|
||||
print('Container startup failed: ' + type(error).__name__, file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,20 @@
|
||||
[
|
||||
{
|
||||
"name": "jre",
|
||||
"version": "21.0.13+9.0.LTS",
|
||||
"url": "https://github.com/adoptium/temurin21-binaries/releases/download/jdk-21.0.13%2B9/OpenJDK21U-jre_x64_linux_hotspot_21.0.13_9.tar.gz",
|
||||
"sha256": "a535a1df9c6284d9618872984e026ae0352d1f7e9e0605425c0e6114f47351dd"
|
||||
},
|
||||
{
|
||||
"name": "python",
|
||||
"version": "3.11.15",
|
||||
"url": "https://github.com/astral-sh/python-build-standalone/releases/download/20260623/cpython-3.11.15%2B20260623-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz",
|
||||
"sha256": "0604cd029b142dc223e131f17f5941c0c8d2d5074997c8178b515b19eea2a6c2"
|
||||
},
|
||||
{
|
||||
"name": "mysql",
|
||||
"version": "8.4.13",
|
||||
"url": "https://cdn.mysql.com/Downloads/MySQL-8.4/mysql-8.4.13-linux-glibc2.28-x86_64-minimal.tar.xz",
|
||||
"sha256": "5dfbb764f731df40ef9e4f5d69aed32bdac8ba1af20bd96b7d40a942d83d4ed2"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,108 @@
|
||||
name: quality-center
|
||||
|
||||
x-logging: &bounded-logging
|
||||
driver: local
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "10"
|
||||
|
||||
services:
|
||||
mysql:
|
||||
image: mysql:8.4.13
|
||||
platform: linux/amd64
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
MYSQL_DATABASE: ${MYSQL_DATABASE:-quality_platform}
|
||||
MYSQL_USER: ${MYSQL_USER:-quality_app}
|
||||
MYSQL_ROOT_HOST: "%"
|
||||
MYSQL_ROOT_PASSWORD_FILE: /run/secrets/mysql_root_password
|
||||
MYSQL_PASSWORD_FILE: /run/secrets/mysql_password
|
||||
TZ: Asia/Shanghai
|
||||
secrets: [mysql_root_password, mysql_password]
|
||||
command:
|
||||
- --skip-log-bin
|
||||
- --mysqlx=OFF
|
||||
- --max-connections=60
|
||||
- --innodb-buffer-pool-size=512M
|
||||
- --innodb-redo-log-capacity=128M
|
||||
volumes: [mysql_data:/var/lib/mysql]
|
||||
networks: [database]
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, "MYSQL_PWD=$$(cat /run/secrets/mysql_root_password) mysql --user=root --batch --skip-column-names -e 'SELECT 1' >/dev/null"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
start_period: 60s
|
||||
stop_grace_period: 2m
|
||||
mem_limit: 2g
|
||||
logging: *bounded-logging
|
||||
|
||||
app:
|
||||
image: quality-center-app:${QUALITY_IMAGE_TAG:-local}
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: deploy/docker/Dockerfile.app
|
||||
platform: linux/amd64
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
mysql: {condition: service_healthy}
|
||||
environment:
|
||||
MYSQL_DATABASE: ${MYSQL_DATABASE:-quality_platform}
|
||||
MYSQL_USER: ${MYSQL_USER:-quality_app}
|
||||
QUALITY_ADMIN_USERNAME: ${QUALITY_ADMIN_USERNAME:-admin}
|
||||
MYSQL_LEGACY_TIMESTAMP_OFFSET_SECONDS: ${MYSQL_LEGACY_TIMESTAMP_OFFSET_SECONDS:-UNCONFIRMED}
|
||||
QUALITY_ANALYSIS_PROTOCOL_V1_ENABLED: ${QUALITY_ANALYSIS_PROTOCOL_V1_ENABLED:-false}
|
||||
QUALITY_ANALYSIS_AUTONOMOUS_ENABLED: ${QUALITY_ANALYSIS_AUTONOMOUS_ENABLED:-false}
|
||||
JAVA_MAX_HEAP: ${JAVA_MAX_HEAP:-2048m}
|
||||
secrets: [mysql_root_password, mysql_password, admin_password]
|
||||
volumes:
|
||||
- state:/srv/quality/.runtime
|
||||
- archives:/srv/quality/runtimedata/archive
|
||||
- backups:/srv/quality/output/backups
|
||||
- logs:/srv/quality/output/logs
|
||||
networks: [database, frontend]
|
||||
read_only: true
|
||||
tmpfs: [/tmp:mode=1777,size=536870912]
|
||||
cap_drop: [ALL]
|
||||
security_opt: [no-new-privileges:true]
|
||||
healthcheck:
|
||||
test: [CMD, /opt/nao/bin/python, deploy/docker/app.py, health]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 6
|
||||
start_period: 15m
|
||||
stop_grace_period: 5m
|
||||
mem_limit: 6g
|
||||
logging: *bounded-logging
|
||||
|
||||
web:
|
||||
image: quality-center-web:${QUALITY_IMAGE_TAG:-local}
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: deploy/docker/Dockerfile.web
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
app: {condition: service_healthy}
|
||||
ports: ["${HTTP_BIND:-127.0.0.1}:${HTTP_PORT:-5280}:8080"]
|
||||
networks: [frontend]
|
||||
healthcheck:
|
||||
test: [CMD, wget, -q, -O, /dev/null, http://127.0.0.1:8080/healthz]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
mem_limit: 256m
|
||||
logging: *bounded-logging
|
||||
|
||||
networks:
|
||||
database: {internal: true}
|
||||
frontend: {}
|
||||
volumes:
|
||||
mysql_data: {}
|
||||
state: {}
|
||||
archives: {}
|
||||
backups: {}
|
||||
logs: {}
|
||||
secrets:
|
||||
mysql_root_password: {environment: QUALITY_SECRET_MYSQL_ROOT}
|
||||
mysql_password: {environment: QUALITY_SECRET_MYSQL_APP}
|
||||
admin_password: {environment: QUALITY_SECRET_ADMIN}
|
||||
@@ -0,0 +1,27 @@
|
||||
"""构建时下载并校验固定 Linux x86_64 运行时,运行阶段不再下载。"""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import tarfile
|
||||
import tempfile
|
||||
import urllib.request
|
||||
|
||||
|
||||
def install(lock, destination):
|
||||
destination = Path(destination)
|
||||
for asset in json.loads(Path(lock).read_text()):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
temporary = Path(temporary); archive = temporary / 'runtime.tar'
|
||||
with urllib.request.urlopen(asset['url'], timeout=120) as source, archive.open('wb') as output:
|
||||
shutil.copyfileobj(source, output)
|
||||
with archive.open('rb') as source: digest = hashlib.file_digest(source, 'sha256').hexdigest()
|
||||
if digest != asset['sha256']: raise ValueError('Runtime checksum mismatch: ' + asset['name'])
|
||||
unpacked = temporary / 'unpacked'; unpacked.mkdir()
|
||||
with tarfile.open(archive) as source: source.extractall(unpacked, filter='data')
|
||||
children = list(unpacked.iterdir())
|
||||
if len(children) != 1 or not children[0].is_dir(): raise ValueError('Unexpected archive layout')
|
||||
shutil.copytree(children[0], destination / asset['name'], symlinks=True)
|
||||
|
||||
|
||||
if __name__ == '__main__': install('/build/assets.lock.json', '/runtimes')
|
||||
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env python3
|
||||
"""统一 Docker Compose 入口;密码只从本机私有文件加载,不进入源码或命令参数。"""
|
||||
import argparse
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import secrets
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
SECRET_NAMES = {'mysql_root_password': 'QUALITY_SECRET_MYSQL_ROOT',
|
||||
'mysql_password': 'QUALITY_SECRET_MYSQL_APP', 'admin_password': 'QUALITY_SECRET_ADMIN'}
|
||||
|
||||
|
||||
def initialize(root=ROOT):
|
||||
base = Path(root) / '.runtime/docker'
|
||||
directory = base / 'secrets'
|
||||
if base.is_symlink() or directory.is_symlink(): raise ValueError('Unsafe private settings path')
|
||||
directory.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
base.chmod(0o700); directory.chmod(0o700)
|
||||
for name in SECRET_NAMES:
|
||||
path = directory / name
|
||||
if path.is_symlink(): raise ValueError('Unsafe secret file')
|
||||
if path.exists(): continue
|
||||
fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
||||
with os.fdopen(fd, 'w') as stream: stream.write(secrets.token_urlsafe(32) + '\n')
|
||||
config = base / 'config.env'
|
||||
if not config.exists():
|
||||
shutil.copyfile(Path(root) / 'deploy/docker/.env.example', config); config.chmod(0o600)
|
||||
return base
|
||||
|
||||
|
||||
def environment(root=ROOT):
|
||||
env = os.environ.copy(); directory = Path(root) / '.runtime/docker/secrets'
|
||||
for name, variable in SECRET_NAMES.items():
|
||||
path = directory / name
|
||||
if path.is_symlink() or not path.is_file() or path.stat().st_mode & 0o077:
|
||||
raise ValueError('Missing or unsafe secret file; run init first')
|
||||
value = path.read_text().strip()
|
||||
if not re.fullmatch(r'[A-Za-z0-9_-]{24,128}', value): raise ValueError('Invalid secret file')
|
||||
env[variable] = value
|
||||
return env
|
||||
|
||||
|
||||
def command(action):
|
||||
actions = {'config': ['config', '--quiet'], 'build': ['build', '--pull'], 'up': ['up', '-d', '--wait', '--wait-timeout', '1200'],
|
||||
'stop': ['stop'], 'status': ['ps'], 'logs': ['logs', '--tail', '100'],
|
||||
'backup': ['exec', '-T', 'app', '/opt/nao/bin/python', 'scripts/backup.py', 'create-and-verify'],
|
||||
'review-retention': ['exec', '-T', 'app', '/opt/nao/bin/python', 'deploy/docker/app.py', 'review-retention']}
|
||||
return ['docker', 'compose', '-f', str(ROOT / 'deploy/docker/compose.yaml'),
|
||||
'--env-file', str(ROOT / '.runtime/docker/config.env'), *actions[action]]
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action', choices=['init', 'config', 'build', 'up', 'stop', 'status', 'logs', 'backup', 'review-retention'])
|
||||
args = parser.parse_args()
|
||||
if args.action == 'init':
|
||||
base = initialize(); print('Private settings prepared: ' + str(base)); return
|
||||
if not shutil.which('docker'): parser.error('Docker Engine and Compose V2 are required')
|
||||
subprocess.run(command(args.action), cwd=ROOT, env=environment(), check=True)
|
||||
|
||||
|
||||
if __name__ == '__main__': main()
|
||||
@@ -0,0 +1,21 @@
|
||||
server {
|
||||
listen 8080;
|
||||
server_name _;
|
||||
root /usr/share/nginx/html;
|
||||
client_max_body_size 260m;
|
||||
resolver 127.0.0.11 valid=10s;
|
||||
set $quality_backend http://app:5281;
|
||||
location = /healthz { access_log off; proxy_pass $quality_backend/actuator/health; }
|
||||
location /api/ {
|
||||
proxy_pass $quality_backend$request_uri;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header Connection '';
|
||||
proxy_buffering off;
|
||||
proxy_read_timeout 1800s;
|
||||
proxy_send_timeout 1800s;
|
||||
}
|
||||
location / { try_files $uri $uri/ /index.html; }
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,148 @@
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path[:0] = [str(ROOT / 'scripts'), str(ROOT / 'deploy/macos')]
|
||||
|
||||
|
||||
def module(name, path):
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
result = importlib.util.module_from_spec(spec); spec.loader.exec_module(result)
|
||||
return result
|
||||
|
||||
|
||||
app = module('container_app', Path(__file__).with_name('app.py'))
|
||||
manage = module('container_manage', Path(__file__).with_name('manage.py'))
|
||||
import backup
|
||||
import backup_retention as br
|
||||
|
||||
|
||||
class ContainerOperationsTests(unittest.TestCase):
|
||||
def test_retirement_rename_stays_on_the_backup_volume_and_resumes_cleanup(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root=Path(directory).resolve();folder=root/'output/backups/20261110T010000Z-12345678'
|
||||
folder.mkdir(parents=True);(folder/'snapshot').write_bytes(b'old backup')
|
||||
keep=folder.parent/'unrelated';keep.mkdir()
|
||||
br.retire_folder(root,folder,'backup')
|
||||
self.assertFalse(folder.exists())
|
||||
record=next((root/'.runtime/retention/retired').glob('*.json'))
|
||||
value=json.loads(record.read_text());self.assertEqual(value['storage'],'backup-volume')
|
||||
self.assertTrue((root/'output/backups/.retired'/value['target']).is_dir())
|
||||
self.assertEqual(br.clean_retired(root),1);self.assertTrue(keep.is_dir())
|
||||
self.assertFalse(record.exists())
|
||||
|
||||
def test_secret_initialization_is_private_idempotent_and_not_in_command_arguments(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory); (root / 'deploy/docker').mkdir(parents=True)
|
||||
(root / 'deploy/docker/.env.example').write_text('HTTP_BIND=127.0.0.1\n')
|
||||
base = manage.initialize(root)
|
||||
before = {p.name: p.read_bytes() for p in (base / 'secrets').iterdir()}
|
||||
manage.initialize(root)
|
||||
self.assertEqual(before, {p.name: p.read_bytes() for p in (base / 'secrets').iterdir()})
|
||||
for path in (base / 'secrets').iterdir(): self.assertEqual(path.stat().st_mode & 0o077, 0)
|
||||
env = manage.environment(root)
|
||||
self.assertEqual(len(set(env[k] for k in manage.SECRET_NAMES.values())), 3)
|
||||
args = manage.command('up')
|
||||
for value in before.values(): self.assertNotIn(value.decode().strip(), str(args))
|
||||
self.assertNotIn('-v', manage.command('stop'))
|
||||
|
||||
def test_config_uses_tcp_and_refuses_uncoordinated_secret_changes(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
with patch.object(app, 'secret', return_value='a' * 32), patch.dict(os.environ, {}, clear=True):
|
||||
first = app.prepare(root)
|
||||
self.assertEqual(first['mysqlHost'], 'mysql'); self.assertFalse(first['binaryLogEnabled'])
|
||||
self.assertEqual((root / '.runtime/local-config.json').stat().st_mode & 0o077, 0)
|
||||
env = app.backend_environment(first)
|
||||
self.assertIn('jdbc:mysql://mysql:3306/', env['MYSQL_URL'])
|
||||
self.assertEqual(env['SERVER_ADDRESS'], '0.0.0.0')
|
||||
with patch.object(app, 'secret', return_value='b' * 32):
|
||||
with self.assertRaises(ValueError): app.prepare(root)
|
||||
|
||||
def test_backend_exit_causes_container_failure_instead_of_leaving_a_dead_api(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory); (root / '.runtime').mkdir()
|
||||
child = Mock(pid=777); child.poll.return_value = 2
|
||||
with patch.object(app, 'ROOT', root), patch.object(app, 'prepare', return_value={}), \
|
||||
patch.object(app, 'backend_environment', return_value={}), patch.object(br, 'accepted_backend'), \
|
||||
patch.object(app.subprocess, 'Popen', return_value=child), patch.object(app.signal, 'signal'):
|
||||
self.assertEqual(app.serve(), 1)
|
||||
child.wait.assert_called_once()
|
||||
|
||||
def test_shutdown_signals_backend_and_workers_before_waiting(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory); (root / '.runtime').mkdir()
|
||||
children = [Mock(pid=100 + i) for i in range(3)]
|
||||
for child in children: child.poll.return_value = None
|
||||
handlers = {}
|
||||
def register(signum, handler): handlers[signum] = handler
|
||||
def tick(_): handlers[app.signal.SIGTERM](app.signal.SIGTERM, None)
|
||||
with patch.object(app, 'ROOT', root), patch.object(app, 'prepare', return_value={}), \
|
||||
patch.object(app, 'backend_environment', return_value={}), patch.object(br, 'accepted_backend'), \
|
||||
patch.object(app.subprocess, 'Popen', side_effect=children), patch.object(app.signal, 'signal', side_effect=register), \
|
||||
patch.object(app, 'ready', return_value=True), patch.object(app, 'review_retention'), \
|
||||
patch.object(app.time, 'sleep', side_effect=tick), patch.object(app.os, 'killpg') as kill:
|
||||
self.assertEqual(app.serve(), 0)
|
||||
self.assertEqual(kill.call_count, 3)
|
||||
for child in children: child.wait.assert_called_once()
|
||||
|
||||
def test_container_pinned_jar_rejects_corruption(self):
|
||||
with tempfile.TemporaryDirectory() as directory, patch.dict(os.environ, {'QUALITY_CONTAINER': '1'}):
|
||||
root = Path(directory); (root / 'app').mkdir()
|
||||
jar = root / 'app/backend.jar'; jar.write_bytes(b'accepted')
|
||||
(root / 'app/backend.sha256').write_text(br.sha(jar))
|
||||
self.assertEqual(br.accepted_backend(root)[0], jar.resolve())
|
||||
jar.write_bytes(b'changed')
|
||||
with self.assertRaises(ValueError): br.accepted_backend(root)
|
||||
|
||||
def test_backup_uses_owned_private_instance_for_container_and_owns_temporary_files(self):
|
||||
from contextlib import contextmanager
|
||||
import backup_verifier
|
||||
configuration = Path('/tmp/owned-verifier/client.cnf')
|
||||
@contextmanager
|
||||
def instance(root): yield configuration
|
||||
seen = []
|
||||
def verify(folder):
|
||||
seen.append((backup.client_config(), backup.verification_temp_root()))
|
||||
return 'verified'
|
||||
with tempfile.TemporaryDirectory() as directory, patch.object(backup, 'ROOT', Path(directory)), \
|
||||
patch.dict(os.environ, {'QUALITY_CONTAINER': '1'}), patch.object(backup_verifier, 'instance', instance), \
|
||||
patch.object(backup, '_verify', side_effect=verify):
|
||||
self.assertEqual(backup.verify('unused'), 'verified')
|
||||
self.assertEqual(seen, [(configuration, configuration.parent)])
|
||||
self.assertIsNone(backup.CLIENT_CONFIG_OVERRIDE.get())
|
||||
|
||||
def test_compose_preserves_volumes_bounded_logs_secrets_and_private_networks(self):
|
||||
try: import yaml
|
||||
except ImportError: self.skipTest('Install PyYAML for Compose contract checks')
|
||||
compose = yaml.safe_load((ROOT / 'deploy/docker/compose.yaml').read_text())
|
||||
services = compose['services']
|
||||
self.assertNotIn('ports', services['mysql']); self.assertNotIn('ports', services['app'])
|
||||
self.assertEqual(services['mysql']['command'][0], '--skip-log-bin')
|
||||
self.assertEqual(len(compose['volumes']), 5)
|
||||
self.assertEqual(services['app']['depends_on']['mysql']['condition'], 'service_healthy')
|
||||
for service in services.values():
|
||||
self.assertEqual(service['restart'], 'unless-stopped')
|
||||
self.assertEqual(service['logging']['options']['max-size'], '10m')
|
||||
self.assertTrue(services['app']['read_only'])
|
||||
self.assertNotIn('docker.sock', json.dumps(compose))
|
||||
self.assertFalse((ROOT / '.gitea/workflows').exists())
|
||||
|
||||
def test_build_context_excludes_live_data_and_private_baseline(self):
|
||||
ignore = (ROOT / '.dockerignore').read_text()
|
||||
self.assertEqual(next(line for line in ignore.splitlines() if line and not line.startswith('#')), '**')
|
||||
for allowed in ('!.runtime', '!runtimedata', '!output', '!质量数据', '!apikey.txt'):
|
||||
self.assertNotIn(allowed, ignore)
|
||||
self.assertIn('**/original-workbook-acceptance.json', ignore)
|
||||
nginx = (ROOT / 'deploy/docker/nginx.conf').read_text()
|
||||
self.assertIn('proxy_buffering off', nginx)
|
||||
self.assertIn('resolver 127.0.0.11', nginx)
|
||||
|
||||
|
||||
if __name__ == '__main__': unittest.main()
|
||||
@@ -0,0 +1,88 @@
|
||||
#!/usr/bin/env python3
|
||||
"""与后端共享 PID 命名空间的低优先级维护任务;不向容器暴露 Docker socket。"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
import shutil
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path[:0] = [str(ROOT / 'scripts'), str(ROOT / 'deploy/macos')]
|
||||
import backup_retention as br
|
||||
import operations
|
||||
import retention
|
||||
|
||||
|
||||
def console_notify(title, message):
|
||||
print(json.dumps({'event': 'capacity-alert', 'title': title, 'message': message}, ensure_ascii=False), flush=True)
|
||||
return {'submitted': True, 'channel': 'container-log'}
|
||||
|
||||
|
||||
def snapshot(root, support=None, now=None):
|
||||
import backup
|
||||
import managed_test_resources as resources
|
||||
import log_retention
|
||||
root = Path(root); now = time.time() if now is None else now
|
||||
cfg = json.loads((root / '.runtime/local-config.json').read_text())
|
||||
rows = backup.client('SELECT TABLE_SCHEMA,COALESCE(SUM(DATA_LENGTH+INDEX_LENGTH),0) FROM information_schema.tables GROUP BY TABLE_SCHEMA;')
|
||||
estimates = {parts[0]: int(parts[1]) for line in rows.splitlines() if len(parts := line.split('\t')) == 2}
|
||||
registered = 0; overdue = 0
|
||||
for path in resources.registry(root).glob('*.json'):
|
||||
item = resources.load(root, path.stem)
|
||||
if not item or not item.get('created'): continue
|
||||
registered += estimates.get(item['database'], 0)
|
||||
if now >= item.get('expiresAt', now + 1): overdue += 1
|
||||
private = sum(operations.allocated(p) for p in (root / '.runtime/retention').glob('verify-*') if p.is_dir() and not p.is_symlink())
|
||||
mysql = estimates.get(cfg['mysqlDatabase'], 0)
|
||||
archive = operations.allocated(root / 'runtimedata/archive'); backups = operations.allocated(root / 'output/backups')
|
||||
native = operations.allocated(root / '.runtime/nao-users')
|
||||
logs = sum(p.stat().st_blocks * 512 for p in log_retention.candidates(root, root / '.runtime/operations'))
|
||||
core = mysql + archive + backups + native + logs
|
||||
free = min(shutil.disk_usage(root / p).free for p in ('.runtime', 'output/backups', 'output/logs', 'runtimedata/archive'))
|
||||
return {'at': now, 'mysqlBytes': mysql, 'mysqlMeasurement': 'information_schema_estimate_excludes_system_redo_undo',
|
||||
'archiveBytes': archive, 'backupBytes': backups, 'logBytes': logs, 'nativeDataBytes': native,
|
||||
'registeredTestBytes': registered, 'privateVerifierBytes': private, 'temporaryBytes': registered + private,
|
||||
'coreBytes': core, 'artifactBytes': 0, 'permanentBytes': core, 'freeBytes': free,
|
||||
'overdueTestResources': overdue}
|
||||
|
||||
|
||||
def once(role):
|
||||
if role == 'monitor':
|
||||
operations.snapshot = snapshot
|
||||
return operations.run(ROOT, support=ROOT / '.runtime/operations', notifier=console_notify)
|
||||
return retention.run(ROOT)
|
||||
|
||||
|
||||
def main():
|
||||
role = sys.argv[1]
|
||||
if role not in ('retention', 'monitor'): raise ValueError('Unknown worker role')
|
||||
try: os.nice(10)
|
||||
except OSError: pass
|
||||
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s')
|
||||
stopping = False
|
||||
def stop(signum, frame):
|
||||
nonlocal stopping
|
||||
stopping = True
|
||||
signal.signal(signal.SIGTERM, stop); signal.signal(signal.SIGINT, stop)
|
||||
while not stopping:
|
||||
try:
|
||||
once(role)
|
||||
except (br.Busy, retention.Busy): pass
|
||||
except Exception as error:
|
||||
if role == 'monitor': operations.failed_run(ROOT, error, notifier=console_notify)
|
||||
else:
|
||||
path = ROOT / '.runtime/retention/state.json'
|
||||
state = json.loads(path.read_text()) if path.exists() else {}
|
||||
state['consecutiveFailures'] = min(1000, state.get('consecutiveFailures', 0) + 1)
|
||||
state['lastError'] = {'time': time.time(), 'type': type(error).__name__}
|
||||
br.atomic(path, state)
|
||||
print(json.dumps({'event': 'maintenance-error', 'role': role, 'errorType': type(error).__name__}), flush=True)
|
||||
for _ in range(300):
|
||||
if stopping: break
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
if __name__ == '__main__': main()
|
||||
Reference in New Issue
Block a user