新增:Linux Docker 部署与自动回收、备份及容量监测
This commit is contained in:
@@ -0,0 +1,94 @@
|
||||
# 本机服务自动启动
|
||||
|
||||
本机使用 `com.lyunlab.quality-center-service` LaunchAgent:解锁并登录当前 macOS 用户后启动,持续守护现有 MySQL 和已验收的后端 JAR。SSH 隧道由已有的 `com.lyunlab.quality-center-tunnel` 独立守护。
|
||||
|
||||
启用 FileVault 的电脑在解锁登录前无法运行用户级守护。退出用户会话、电脑关机、休眠或断网期间,不能依靠这台 Mac 持续提供公网服务。需要全天候服务时,按 [Docker 部署说明](../docker/README.md) 将后台和数据库部署到常在线服务器。
|
||||
|
||||
## 管理
|
||||
|
||||
首次安装维护环境时运行 `python3 -m venv .runtime/retention-env`,再使用 `.runtime/retention-env/bin/pip install --require-hashes -r deploy/macos/retention-requirements.txt` 安装固定依赖。该步骤只安装维护工具,不初始化业务数据库。
|
||||
|
||||
在项目目录执行:
|
||||
|
||||
```sh
|
||||
python3 deploy/macos/service.py status
|
||||
python3 deploy/macos/service.py pause
|
||||
python3 deploy/macos/service.py stop
|
||||
python3 deploy/macos/service.py start
|
||||
```
|
||||
|
||||
- `status`:显示守护、进程身份、数据库和接口就绪状态。
|
||||
- `pause`:停用自动启动和守护,保留当前 MySQL / 后端进程,适合维护前使用。
|
||||
- `stop`:停用守护,按后端、MySQL 的顺序正常停服;超时不会强制结束数据库。
|
||||
- `start`:重新启用自动启动和守护,先确认数据库可查询,再启动后端。
|
||||
|
||||
维护期间先暂停守护,再操作服务。直接运行旧的停止脚本后,仍启用的守护会把进程重新拉起。
|
||||
|
||||
## 实现与边界
|
||||
|
||||
- 安装时登记已验收 JAR 的 SHA-256,从 `.runtime/backend-builds/` 启动对应的不可变运行包。
|
||||
- 只使用 `.runtime/local-config.json` 指定的已初始化数据库;缺少数据标记或运行包摘要不符时停止启动,不初始化、不导入数据库。
|
||||
- 每 3 秒检查进程;异常退出后等待 10 秒重启。数据库可查询后才启动后端。端口被未知进程占用或进程身份无法确认时保留现场。
|
||||
- 守护自身由 launchd 的 `KeepAlive` 负责重启。重新启动守护时会核对启动时间、用户、程序和参数,接管仍存活的服务,避免重复启动。
|
||||
- 只对已经退出的进程自动重启。健康检查暂时失败时不强行终止正在导入或查询的数据库。
|
||||
- 守护脚本按摘要复制到用户的 Application Support 目录。项目源文件的后续编辑不会立即改变已安装脚本。
|
||||
- MySQL 的 `binaryLogEnabled` 配置为 false 时,使用 `--skip-log-bin` 启动。事务重做日志和 undo 文件继续保留。历史时间点恢复不可用,恢复依赖最新完整快照。
|
||||
|
||||
启动配置:`~/Library/LaunchAgents/com.lyunlab.quality-center-service.plist`。
|
||||
|
||||
状态和滚动日志:`~/Library/Application Support/QualityCenterService/`。
|
||||
|
||||
进程业务日志:项目的 `output/logs/`。
|
||||
|
||||
重新安装或升级守护脚本时使用 `service.py install --root <项目绝对路径> --jar <已验收JAR绝对路径>`。部署配置中只记录路径与摘要,密码仍从原有私有配置读取。
|
||||
|
||||
## 验证
|
||||
|
||||
```sh
|
||||
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s deploy/macos -p 'test_service.py' -v
|
||||
```
|
||||
|
||||
隔离进程检查覆盖 MySQL 与后端异常退出重启、数据库启动顺序、已有进程接管、未知进程保护和运行包/数据目录前置校验。实际重启电脑和 FileVault 解锁后的启动仍需单独验收。
|
||||
|
||||
## 自动数据保留
|
||||
|
||||
`com.lyunlab.quality-center-retention` 是独立的低优先级维护任务,解锁登录后运行,每5分钟检查一次。它使用当前已验收 JAR,独立运行维护脚本;不重新构建业务后端。
|
||||
|
||||
- **旧版本**:当前正式版本、候选版本及基线、待配置来源的基线、分析会话、已保存报表/快照/清单、巡检等业务引用均保护。其他已发布版本连续30分钟未被引用,且当前版本已有恢复验证通过的备份后,才进入回收。
|
||||
- **旧构建**:只处理已无版本引用的终态构建。分批清理,每批最多1000行;先撤销其可复用状态和唯一构建键,避免新导入复用正在清理的半成品。仍有未知业务引用时保留。
|
||||
- **旧来源修订**:每5分钟检查,已完成修订连续30分钟没有任何版本、构建、明细、来源清单、替换/撤销/恢复记录或保存内容引用后才回收。修订身份变化或重新出现引用会重置保护期;未知状态、缺失或校验异常的文件保留。SHA-256、大小、路径边界与文件身份全部通过后才删除修订记录,原件另行核对共享路径和保存内容引用。共享文件保留;文件删除意图先持久登记,意外退出后可续清,不扫描删除未登记的孤立文件。
|
||||
- **上传草稿**:24小时过期后再留30分钟保护期。已入库文件先核对正式归档摘要,再重绑定来源路径、清理临时副本;待导入来源仍引用的文件保留。无数据库记录的上传临时文件超过24小时才检查回收。
|
||||
- **备份**:检测到新正式版本或距离上次备份超过24小时,在任务空闲时生成新备份。新备份必须通过独立 MySQL 实例恢复、表行数、正式数据汇总、应用文件和来源归档校验,才原子更新最新备份入口并回收旧备份。失败时保留旧有效备份,失败重试间隔至少1小时。
|
||||
- **任务保护**:导入、数据工作流、分析、报告、巡检、排队任务及待自动重试任务存在时,自动维护延期。每一批删除都重新检查任务状态和当前版本;短暂取得数据库排他访问失败时延期,不终止业务查询或任务。
|
||||
- **互斥与失败恢复**:手动备份、恢复和自动维护共享 `.runtime/data-maintenance.lock`。恢复校验使用临时、仅本机私有 socket 可访问的独立 MySQL,完成后正常关闭并删除;异常遗留只根据所有者记录及进程身份处理,不按数据库名前缀批量删除。
|
||||
- **维护暂停**:`service.py pause/stop` 同时写入维护暂停标记;`service.py start/install` 解除。执行中的备份允许安全完成,后续删除批次会停下。
|
||||
- **升级保护**:当前 JAR 摘要或数据库结构发生变化时暂停自动清理,重新检查兼容性后执行维护安装命令。未知任务状态也按进行中处理。
|
||||
- **空间余量**:开始新备份前预留至少三倍数据库表/索引容量、归档空间和额外安全余量;空间不足时保留现有有效备份并延期。失败的新候选只清理本次登记的文件,未知目录不会删除。
|
||||
|
||||
最新备份记录在 `.runtime/retention/latest.json`,实际目录在 `output/backups/`,界面“任务与监控”的备份列表可查看。
|
||||
|
||||
维护任务状态与滚动日志分别在 `.runtime/retention/state.json` 和 `~/Library/Application Support/QualityCenterService/logs/retention.log`。
|
||||
|
||||
来源修订回收计数为 `sourceRevisionsRemoved`、`sourceArchiveFilesRemoved` 和 `sourceArchiveBytesRemoved`,待解除文件引用的登记数为 `sourceFilesPending`。登记目录为 `.runtime/retention/source-files/`。来源清单中仍可重新导入的记录(包括已导入状态)属于有效引用,不能作为无引用修订删除。
|
||||
|
||||
```sh
|
||||
.runtime/retention-env/bin/python3 deploy/macos/retention.py status --root "$PWD"
|
||||
.runtime/retention-env/bin/python3 deploy/macos/retention.py install --root "$PWD"
|
||||
PYTHONDONTWRITEBYTECODE=1 .runtime/retention-env/bin/python3 -m unittest discover -s deploy/macos -p 'test_retention.py' -v
|
||||
```
|
||||
|
||||
维护依赖 PyMySQL 固定版本及 SHA-256 在 `retention-requirements.txt` 中。私有 Python 环境与依赖不修改全局安装。日常删除释放的 InnoDB 表内空间由后续导入复用;必要的物理表空间整理另在低峰进行。
|
||||
|
||||
## 容量、测试资源与日志
|
||||
|
||||
`operations.py` 安装独立的 5 分钟检查任务。容量阈值为 30/50GiB,余量阈值为 50/20GiB;监测业务数据、交付产物、临时校验空间及约 24 小时增长,保留 30 天日报。告警通过 macOS 系统通知发送,并记录在 `.runtime/operations/`,相同告警 6 小时内去重。
|
||||
|
||||
注册过的测试数据库在正常退出时回收;异常遗留超过 24 小时,确认所有者和子进程退出、标记一致且无连接后才删除。未登记数据库继续保留。日志按 7 天、每类 100MiB 限制;Java 原生日志滚动配置在 `logback-quality.xml`,MySQL 通过重新打开错误日志完成滚动,不重启数据库,不截断活动文件。
|
||||
|
||||
```sh
|
||||
.runtime/retention-env/bin/python3 deploy/macos/operations.py install --root "$PWD"
|
||||
.runtime/retention-env/bin/python3 deploy/macos/operations.py status --root "$PWD"
|
||||
.runtime/retention-env/bin/python3 deploy/macos/operations.py notify-test --root "$PWD"
|
||||
```
|
||||
|
||||
安装日志配置后,后端需在任务空闲时正常重启一次才能加载;不要为加载日志配置中断导入。Linux 复用同一回收逻辑,采用容器日志告警,见 Docker 说明。
|
||||
@@ -0,0 +1,19 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<configuration>
|
||||
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
|
||||
<property name="QUALITY_LOG_DIR" value="${QUALITY_LOG_DIR:-output/logs}"/>
|
||||
<appender name="QUALITY_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
|
||||
<file>${QUALITY_LOG_DIR}/backend.log</file>
|
||||
<append>true</append>
|
||||
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
|
||||
<fileNamePattern>${QUALITY_LOG_DIR}/backend.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
|
||||
<maxFileSize>10MB</maxFileSize>
|
||||
<maxHistory>7</maxHistory>
|
||||
<totalSizeCap>80MB</totalSizeCap>
|
||||
<cleanHistoryOnStart>true</cleanHistoryOnStart>
|
||||
</rollingPolicy>
|
||||
<encoder><pattern>${FILE_LOG_PATTERN}</pattern><charset>UTF-8</charset></encoder>
|
||||
</appender>
|
||||
<logger name="org.springframework.security" level="WARN"/>
|
||||
<root level="INFO"><appender-ref ref="QUALITY_FILE"/></root>
|
||||
</configuration>
|
||||
@@ -0,0 +1,276 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Independent capacity monitoring, registered-test cleanup and log retention."""
|
||||
import argparse
|
||||
import fcntl
|
||||
from datetime import datetime
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
from logging.handlers import RotatingFileHandler
|
||||
import os
|
||||
from pathlib import Path
|
||||
import plistlib
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
LABEL = 'com.lyunlab.quality-center-operations'
|
||||
SUPPORT = Path.home() / 'Library/Application Support/QualityCenterService'
|
||||
GIB = 1024**3
|
||||
DEFAULTS = {'intervalSeconds': 300, 'warningBytes': 30 * GIB, 'criticalBytes': 50 * GIB,
|
||||
'freeWarningBytes': 50 * GIB, 'freeCriticalBytes': 20 * GIB,
|
||||
'growthBytes': 2 * GIB, 'growthFraction': .2, 'notificationCooldownSeconds': 21600,
|
||||
'logMaxAgeSeconds': 604800, 'logClassMaxBytes': 100 * 1024**2,
|
||||
'testMaxAgeSeconds': 86400}
|
||||
|
||||
|
||||
def allocated(path):
|
||||
path = Path(path)
|
||||
if not path.exists() or path.is_symlink(): return 0
|
||||
result = subprocess.run(['/usr/bin/du', '-sk', str(path)], capture_output=True, text=True, timeout=45)
|
||||
lines = result.stdout.splitlines()
|
||||
if result.returncode or not lines: raise RuntimeError('Capacity scan incomplete')
|
||||
return int(lines[-1].split()[0]) * 1024
|
||||
|
||||
|
||||
def snapshot(root, support=SUPPORT, now=None):
|
||||
import managed_test_resources as resources
|
||||
import log_retention
|
||||
root = Path(root).resolve(); now = time.time() if now is None else now
|
||||
mysql = allocated(root / '.runtime/mysql-data')
|
||||
archives = allocated(root / 'runtimedata/archive'); backups = allocated(root / 'output/backups')
|
||||
runtime = allocated(root / '.runtime'); outputs = allocated(root / 'output')
|
||||
logs = log_retention.candidates(root, support)
|
||||
log_bytes = sum(p.stat().st_blocks * 512 for p in logs)
|
||||
output_logs = sum(p.stat().st_blocks * 512 for p in logs if p.is_relative_to(root / 'output'))
|
||||
runtime_logs = sum(p.stat().st_blocks * 512 for p in logs if p.is_relative_to(root / '.runtime'))
|
||||
native = allocated(root / '.runtime/nao-users')
|
||||
registered = 0; overdue = 0
|
||||
for p in resources.registry(root).glob('*.json'):
|
||||
value = resources.load(root, p.stem)
|
||||
if not value or not value.get('created'): continue
|
||||
if now >= value.get('expiresAt', now + 1): overdue += 1
|
||||
registered += allocated(root / '.runtime/mysql-data' / value['database'])
|
||||
private = 0
|
||||
for p in (root / '.runtime/retention').glob('verify-*'):
|
||||
if p.is_symlink(): continue
|
||||
try: owner = json.loads((p / 'owner.json').read_text())
|
||||
except (OSError, ValueError): continue
|
||||
if owner.get('kind') == 'private-quality-restore' and owner.get('directory') == str(p): private += allocated(p)
|
||||
# Core service data is the same scope as the earlier 11.86GiB measurement.
|
||||
# Software/build/delivery artifacts are visible separately and included in growth checks.
|
||||
core = max(0, mysql - registered) + archives + backups + log_bytes + native
|
||||
artifacts = max(0, runtime - mysql - private - native - runtime_logs) + max(0, outputs - backups - output_logs)
|
||||
return {'at': now, 'mysqlBytes': mysql, 'archiveBytes': archives, 'backupBytes': backups,
|
||||
'logBytes': log_bytes, 'nativeDataBytes': native, 'registeredTestBytes': registered,
|
||||
'privateVerifierBytes': private, 'temporaryBytes': registered + private,
|
||||
'coreBytes': core, 'artifactBytes': artifacts, 'permanentBytes': core + artifacts,
|
||||
'freeBytes': shutil.disk_usage(root).free, 'overdueTestResources': overdue}
|
||||
|
||||
|
||||
def conditions(sample, policy, maintenance, history):
|
||||
alerts = {}
|
||||
if sample['coreBytes'] >= policy['criticalBytes']:
|
||||
alerts['capacity'] = {'severity': 'critical', 'message': f"业务数据占用已达{sample['coreBytes']/GIB:.1f}GiB,请检查增长来源。"}
|
||||
elif sample['coreBytes'] >= policy['warningBytes']:
|
||||
alerts['capacity'] = {'severity': 'warning', 'message': f"业务数据占用已达{sample['coreBytes']/GIB:.1f}GiB。"}
|
||||
for key, label, size in [('temporary', '测试及校验临时空间', sample.get('temporaryBytes', 0)),
|
||||
('artifacts', '交付及构建产物', sample.get('artifactBytes', max(0, sample['permanentBytes'] - sample['coreBytes'])))]:
|
||||
if size >= policy['warningBytes']:
|
||||
alerts[key] = {'severity': 'critical' if size >= policy['criticalBytes'] else 'warning',
|
||||
'message': f'{label}已占用{size/GIB:.1f}GiB,请核对;正在运行的任务会继续保护。'}
|
||||
if sample['freeBytes'] < policy['freeCriticalBytes']:
|
||||
alerts['disk'] = {'severity': 'critical', 'message': f"磁盘仅余{sample['freeBytes']/GIB:.1f}GiB。"}
|
||||
elif sample['freeBytes'] < policy['freeWarningBytes']:
|
||||
alerts['disk'] = {'severity': 'warning', 'message': f"磁盘剩余{sample['freeBytes']/GIB:.1f}GiB,请预留备份空间。"}
|
||||
if maintenance.get('consecutiveFailures', 0) >= 3:
|
||||
alerts['maintenance'] = {'severity': 'critical', 'message': '数据维护已连续失败3次,请检查维护记录。'}
|
||||
if sample['overdueTestResources']:
|
||||
alerts['tests'] = {'severity': 'warning', 'message': f"有{sample['overdueTestResources']}个测试资源已超过24小时;正在运行或身份未确认的资源将继续保留。"}
|
||||
old = [v for v in history if 21 * 3600 <= sample['at'] - v['at'] <= 27 * 3600]
|
||||
if old:
|
||||
baseline = min(old, key=lambda x: abs(sample['at'] - x['at'] - 86400))
|
||||
changes = []
|
||||
for label, key in [('业务数据', 'coreBytes'), ('交付及构建产物', 'artifactBytes')]:
|
||||
before = baseline.get(key, baseline['permanentBytes'] - baseline['coreBytes'])
|
||||
after = sample.get(key, sample['permanentBytes'] - sample['coreBytes'])
|
||||
delta = after - before
|
||||
if delta > max(policy['growthBytes'], before * policy['growthFraction']):
|
||||
changes.append(f'{label}增长{delta/GIB:.1f}GiB')
|
||||
if changes:
|
||||
alerts['growth'] = {'severity': 'warning', 'message': '约24小时' + ','.join(changes) + ',请核对。'}
|
||||
return alerts
|
||||
|
||||
|
||||
def notify(title, message):
|
||||
script = 'on run argv\ndisplay notification (item 2 of argv) with title (item 1 of argv)\nend run\n'
|
||||
try:
|
||||
p = subprocess.run(['/usr/bin/osascript', '-', title, message], input=script, text=True,
|
||||
capture_output=True, timeout=10)
|
||||
return {'submitted': p.returncode == 0, 'channel': 'macOS', 'exitCode': p.returncode}
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return {'submitted': False, 'channel': 'macOS'}
|
||||
|
||||
|
||||
def dispatch(alerts, previous, now, policy, notifier=notify):
|
||||
updated = {}; events = []
|
||||
for key, value in alerts.items():
|
||||
prior = previous.get(key, {})
|
||||
due = (not prior or not prior.get('lastSubmittedAt') or prior.get('severity') != value['severity'] or
|
||||
now - prior.get('lastSubmittedAt', 0) >= policy['notificationCooldownSeconds'])
|
||||
entry = {**value, 'since': prior.get('since', now), 'lastSubmittedAt': prior.get('lastSubmittedAt', 0)}
|
||||
if due:
|
||||
result = notifier('质量平台容量告警', value['message'])
|
||||
entry['notification'] = result
|
||||
# Failed submission retries on the next check; duplicates are limited after success.
|
||||
if result.get('submitted'): entry['lastSubmittedAt'] = now
|
||||
events.append({'kind': 'alert', 'key': key, **value, 'at': now, 'notification': result})
|
||||
updated[key] = entry
|
||||
for key in set(previous) - set(alerts):
|
||||
events.append({'kind': 'resolved', 'key': key, 'at': now})
|
||||
return updated, events
|
||||
|
||||
|
||||
def janitor(root, support, policy):
|
||||
import backup_retention as br
|
||||
import backup_verifier
|
||||
import managed_test_resources as resources
|
||||
import log_retention
|
||||
from retention import Database, Busy
|
||||
root = Path(root).resolve()
|
||||
if (root / '.runtime/retention/paused').exists(): return {'deferred': 'Maintenance is paused'}
|
||||
try:
|
||||
with br.lock(root, timeout=0):
|
||||
db = Database(root)
|
||||
try:
|
||||
config = json.loads((root / '.runtime/retention/config.json').read_text())
|
||||
accepted = br.accepted_backend(root)
|
||||
if config['schemaSignature'] != db.schema_signature or not accepted or config['acceptedJarSha256'] != accepted[1]:
|
||||
return {'deferred': 'Application version changed'}
|
||||
db.current()
|
||||
if db.busy(): return {'deferred': 'Application jobs are active'}
|
||||
# Use a short-lived CLI connection so no observer holds a test schema open.
|
||||
import backup
|
||||
result = resources.cleanup(root, backup.client)
|
||||
result['abandonedVerifiersRemoved'] = backup_verifier.clean_abandoned(root)
|
||||
result['mysqlLogsRotated'] = log_retention.rotate_mysql(root, backup.client, br)
|
||||
result.update(log_retention.retain(root, support,
|
||||
max_bytes=policy['logClassMaxBytes'], max_age=policy['logMaxAgeSeconds']))
|
||||
return result
|
||||
finally: db.conn.close()
|
||||
except (br.Busy, Busy): return {'deferred': 'Another maintenance or application operation is active'}
|
||||
|
||||
|
||||
def run(root, support=SUPPORT, notifier=notify):
|
||||
import backup_retention as br
|
||||
root = Path(root).resolve(); base = root / '.runtime/operations'
|
||||
if base.is_symlink() or (root / '.runtime').is_symlink(): raise ValueError('Unsafe operations state path')
|
||||
base.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
policy = {**DEFAULTS, **json.loads((base / 'policy.json').read_text())}
|
||||
state_path = base / 'monitor.json'; previous = json.loads(state_path.read_text()) if state_path.exists() else {}
|
||||
history_path = base / 'samples.json'; history = json.loads(history_path.read_text()) if history_path.exists() else []
|
||||
now = time.time(); sample = snapshot(root, support, now)
|
||||
maintenance_path = root / '.runtime/retention/state.json'
|
||||
maintenance = json.loads(maintenance_path.read_text()) if maintenance_path.exists() else {}
|
||||
alerts = conditions(sample, policy, maintenance, history)
|
||||
cleanup = {}; failures = previous.get('cleanupFailures', 0)
|
||||
try:
|
||||
cleanup = janitor(root, support, policy)
|
||||
if 'deferred' not in cleanup: failures = 0
|
||||
except Exception as error:
|
||||
failures += 1; cleanup = {'errorType': type(error).__name__}
|
||||
if failures >= 3: alerts['cleanup'] = {'severity': 'warning', 'message': '日志或测试环境清理已连续失败3次,请检查维护记录。'}
|
||||
active, events = dispatch(alerts, previous.get('alerts', {}), now, policy, notifier)
|
||||
state = {'checkedAt': now, 'sample': sample, 'cleanup': cleanup, 'cleanupFailures': failures, 'alerts': active}
|
||||
br.atomic(state_path, state)
|
||||
(base / 'failure.json').unlink(missing_ok=True)
|
||||
history = [v for v in history if now - v['at'] <= 27 * 3600]
|
||||
history.append(sample); br.atomic(history_path, history[-330:])
|
||||
daily_path = base / 'daily.json'; daily = json.loads(daily_path.read_text()) if daily_path.exists() else {}
|
||||
day = datetime.fromtimestamp(now, ZoneInfo('Asia/Shanghai')).date().isoformat(); daily[day] = sample
|
||||
br.atomic(daily_path, {k: daily[k] for k in sorted(daily)[-30:]})
|
||||
if events:
|
||||
handler = RotatingFileHandler(base / 'alerts.jsonl', maxBytes=1024**2, backupCount=2)
|
||||
handler.setFormatter(logging.Formatter('%(message)s'))
|
||||
logger = logging.getLogger('quality-operations-alerts'); logger.addHandler(handler); logger.setLevel(logging.INFO); logger.propagate = False
|
||||
try:
|
||||
for event in events: logger.info(json.dumps(event, ensure_ascii=False))
|
||||
finally: logger.removeHandler(handler); handler.close()
|
||||
return state
|
||||
|
||||
|
||||
def failed_run(root, error, notifier=notify):
|
||||
import backup_retention as br
|
||||
base = Path(root).resolve() / '.runtime/operations'
|
||||
if base.is_symlink() or base.parent.is_symlink(): return
|
||||
path = base / 'failure.json'; value = json.loads(path.read_text()) if path.exists() else {}
|
||||
now = time.time(); value.update({'count': value.get('count', 0) + 1, 'time': now, 'type': type(error).__name__})
|
||||
if value['count'] >= 3 and now - value.get('lastSubmittedAt', 0) >= DEFAULTS['notificationCooldownSeconds']:
|
||||
result = notifier('质量平台监测异常', '容量监测已连续失败3次,请查看监测状态。')
|
||||
value['notification'] = result
|
||||
if result.get('submitted'): value['lastSubmittedAt'] = now
|
||||
br.atomic(path, value)
|
||||
|
||||
|
||||
def install(root):
|
||||
import backup_retention as br
|
||||
root = Path(root).resolve(); base = root / '.runtime/operations'
|
||||
base.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
policy_path = base / 'policy.json'
|
||||
policy = {**DEFAULTS, **(json.loads(policy_path.read_text()) if policy_path.exists() else {})}
|
||||
br.atomic(policy_path, policy)
|
||||
files = [Path(__file__).resolve(), root / 'deploy/macos/retention.py'] + [root / 'scripts' / p for p in
|
||||
('backup_retention.py', 'revision_retention.py', 'backup_verifier.py', 'managed_test_resources.py', 'log_retention.py')]
|
||||
digest = hashlib.sha256(b''.join(p.read_bytes() for p in files)).hexdigest()
|
||||
version = SUPPORT / 'operations' / digest; version.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
for p in files: shutil.copy2(p, version / p.name); (version / p.name).chmod(0o600)
|
||||
log_config = root / 'deploy/macos/logback-quality.xml'
|
||||
shutil.copy2(log_config, base / 'logback-quality.xml'); (base / 'logback-quality.xml').chmod(0o600)
|
||||
logs = SUPPORT / 'logs'; logs.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
value = {'Label': LABEL, 'ProgramArguments': [str(root / '.runtime/retention-env/bin/python3'), str(version / 'operations.py'), 'run', '--root', str(root)],
|
||||
'RunAtLoad': True, 'StartInterval': 300, 'WorkingDirectory': str(root), 'ProcessType': 'Background',
|
||||
'Nice': 10, 'LowPriorityIO': True, 'EnvironmentVariables': {'PYTHONDONTWRITEBYTECODE': '1'},
|
||||
'StandardOutPath': str(logs / 'operations-launchd.log'), 'StandardErrorPath': str(logs / 'operations-launchd-error.log')}
|
||||
domain = 'gui/' + str(os.getuid()); plist = Path.home() / 'Library/LaunchAgents' / (LABEL + '.plist')
|
||||
subprocess.run(['launchctl', 'bootout', domain + '/' + LABEL], capture_output=True)
|
||||
with plist.open('wb') as f: plistlib.dump(value, f)
|
||||
plist.chmod(0o600); subprocess.run(['launchctl', 'enable', domain + '/' + LABEL], check=True)
|
||||
for _ in range(50):
|
||||
p = subprocess.run(['launchctl', 'bootstrap', domain, str(plist)], capture_output=True)
|
||||
if p.returncode == 0: break
|
||||
time.sleep(.2)
|
||||
else: raise RuntimeError('Operations job registration did not complete')
|
||||
print('Installed independent five-minute capacity and housekeeping checks')
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action', choices=['install', 'run', 'status', 'notify-test']); parser.add_argument('--root', required=True)
|
||||
args = parser.parse_args(); root = Path(args.root).resolve(); sys.path.append(str(root / 'scripts'))
|
||||
import backup_retention as br
|
||||
if args.action == 'install':
|
||||
with br.lock(root, timeout=0): install(root)
|
||||
elif args.action == 'status':
|
||||
print((root / '.runtime/operations/monitor.json').read_text())
|
||||
elif args.action == 'notify-test':
|
||||
print(json.dumps(notify('质量平台通知测试', '容量监测已启用。这是测试通知,当前没有容量告警。'), ensure_ascii=False))
|
||||
else:
|
||||
logs = SUPPORT / 'logs'; logs.mkdir(parents=True, exist_ok=True)
|
||||
handler = RotatingFileHandler(logs / 'operations.log', maxBytes=2 * 1024**2, backupCount=2)
|
||||
handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %(message)s'))
|
||||
logging.getLogger().addHandler(handler); logging.getLogger().setLevel(logging.INFO)
|
||||
with (root / '.runtime/operations/monitor.lock').open('a') as monitor_lock:
|
||||
fcntl.flock(monitor_lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
try:
|
||||
state = run(root)
|
||||
logging.info('operations complete coreGiB=%.3f temporaryGiB=%.3f artifactsGiB=%.3f cleanup=%s alerts=%s',
|
||||
state['sample']['coreBytes'] / GIB, state['sample']['temporaryBytes'] / GIB,
|
||||
state['sample']['artifactBytes'] / GIB, json.dumps(state['cleanup']), list(state['alerts']))
|
||||
except Exception as error:
|
||||
logging.error('operations failed safely: %s', type(error).__name__)
|
||||
failed_run(root, error); raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__': main()
|
||||
@@ -0,0 +1,23 @@
|
||||
PyMySQL==1.2.3 --hash=sha256:14f1c68e2ed859243ae5ca41ffbe677027fc46bc136a9f0be8a4e928e5e7415a
|
||||
psutil==7.2.2 \
|
||||
--hash=sha256:0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372 \
|
||||
--hash=sha256:076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9 \
|
||||
--hash=sha256:11fe5a4f613759764e79c65cf11ebdf26e33d6dd34336f8a337aa2996d71c841 \
|
||||
--hash=sha256:1a571f2330c966c62aeda00dd24620425d4b0cc86881c89861fbc04549e5dc63 \
|
||||
--hash=sha256:1a7b04c10f32cc88ab39cbf606e117fd74721c831c98a27dc04578deb0c16979 \
|
||||
--hash=sha256:1fa4ecf83bcdf6e6c8f4449aff98eefb5d0604bf88cb883d7da3d8d2d909546a \
|
||||
--hash=sha256:2edccc433cbfa046b980b0df0171cd25bcaeb3a68fe9022db0979e7aa74a826b \
|
||||
--hash=sha256:7b6d09433a10592ce39b13d7be5a54fbac1d1228ed29abc880fb23df7cb694c9 \
|
||||
--hash=sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee \
|
||||
--hash=sha256:917e891983ca3c1887b4ef36447b1e0873e70c933afc831c6b6da078ba474312 \
|
||||
--hash=sha256:ab486563df44c17f5173621c7b198955bd6b613fb87c71c161f827d3fb149a9b \
|
||||
--hash=sha256:ae0aefdd8796a7737eccea863f80f81e468a1e4cf14d926bd9b6f5f2d5f90ca9 \
|
||||
--hash=sha256:b0726cecd84f9474419d67252add4ac0cd9811b04d61123054b9fb6f57df6e9e \
|
||||
--hash=sha256:b58fabe35e80b264a4e3bb23e6b96f9e45a3df7fb7eed419ac0e5947c61e47cc \
|
||||
--hash=sha256:c7663d4e37f13e884d13994247449e9f8f574bc4655d509c3b95e9ec9e2b9dc1 \
|
||||
--hash=sha256:e452c464a02e7dc7822a05d25db4cde564444a67e58539a00f929c51eddda0cf \
|
||||
--hash=sha256:e78c8603dcd9a04c7364f1a3e670cea95d51ee865e4efb3556a3a63adef958ea \
|
||||
--hash=sha256:eb7e81434c8d223ec4a219b5fc1c47d0417b12be7ea866e24fb5ad6e84b3d988 \
|
||||
--hash=sha256:ed0cace939114f62738d808fdcecd4c869222507e266e574799e9c0faa17d486 \
|
||||
--hash=sha256:eed63d3b4d62449571547b60578c5b2c4bcccc5387148db46e0c2313dad0ee00 \
|
||||
--hash=sha256:fd04ef36b4a6d599bbdb225dd1d3f51e00105f6d48a28f006da7f9822f2606d8
|
||||
@@ -0,0 +1,525 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Conservative, online retention for this Mac's accepted Quality Center runtime."""
|
||||
import argparse
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime
|
||||
import hashlib
|
||||
import importlib
|
||||
import json
|
||||
import logging
|
||||
from logging.handlers import RotatingFileHandler
|
||||
import os
|
||||
from pathlib import Path
|
||||
import plistlib
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
import pymysql
|
||||
from pymysql.cursors import DictCursor
|
||||
|
||||
LABEL = 'com.lyunlab.quality-center-retention'
|
||||
SUPPORT = Path.home() / 'Library/Application Support/QualityCenterService'
|
||||
TERMINAL = ('COMPLETED', 'FAILED', 'CANCELLED', 'SUCCEEDED', 'STOPPED', 'REJECTED', 'CLOSED', 'INTERRUPTED', 'INVALIDATED')
|
||||
BUILD_CHILDREN = ('defect_component', 'fine_defect_rollup', 'metric_rollup', 'sales_order_metric_rollup',
|
||||
'fine_defect_rollup_build', 'metric_rollup_build', 'sales_order_metric_rollup_build',
|
||||
'fact_black', 'fact_bright')
|
||||
OWNED_RELEASE = ('release_member', 'release_protocol_dependency', 'release_source_exclusion',
|
||||
'release_action', 'candidate_reconciliation')
|
||||
MUTABLE_RELEASE_REFS = ('data_release', 'import_run', 'release_action', 'candidate_reconciliation')
|
||||
|
||||
|
||||
def ident(value):
|
||||
if not re.fullmatch(r'[A-Za-z0-9_]+', value):
|
||||
raise ValueError('Invalid SQL identifier')
|
||||
return '`' + value + '`'
|
||||
|
||||
|
||||
class Busy(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def safe_file(path, boundary):
|
||||
path, boundary = Path(path), Path(boundary).absolute()
|
||||
if not path.is_absolute():
|
||||
return False
|
||||
if not path.is_relative_to(boundary) or not path.resolve().is_relative_to(boundary.resolve()):
|
||||
return False
|
||||
cursor = path
|
||||
while cursor != boundary:
|
||||
if cursor.is_symlink():
|
||||
return False
|
||||
cursor = cursor.parent
|
||||
return not boundary.is_symlink() and path.is_file()
|
||||
|
||||
|
||||
def fingerprint(path):
|
||||
s = Path(path).stat()
|
||||
return (s.st_dev, s.st_ino, s.st_size, s.st_mtime_ns)
|
||||
|
||||
|
||||
class Database:
|
||||
def __init__(self, root, database=None):
|
||||
self.root = Path(root).resolve()
|
||||
self.config = json.loads((self.root / '.runtime/local-config.json').read_text())
|
||||
self.name = database or self.config['mysqlDatabase']
|
||||
ident(self.name)
|
||||
endpoint = ({'host': self.config['mysqlHost'], 'port': int(self.config['mysqlPort'])}
|
||||
if self.config.get('mysqlHost') else {'unix_socket': str(self.root / '.runtime/mysql.sock')})
|
||||
self.conn = pymysql.connect(**endpoint, user='root',
|
||||
password=self.config['rootPassword'], database=self.name, charset='utf8mb4',
|
||||
autocommit=True, cursorclass=DictCursor, connect_timeout=3, read_timeout=10, write_timeout=5)
|
||||
self.execute("SET SESSION time_zone='+00:00'")
|
||||
self.execute('SET SESSION lock_wait_timeout=1')
|
||||
self.execute('SET SESSION innodb_lock_wait_timeout=1')
|
||||
rows = self.rows('SELECT TABLE_NAME,COLUMN_NAME,IS_NULLABLE,DATA_TYPE FROM information_schema.columns WHERE TABLE_SCHEMA=%s', (self.name,))
|
||||
self.columns = {}
|
||||
self.column_types = {}
|
||||
for row in rows:
|
||||
self.columns.setdefault(row['TABLE_NAME'], {})[row['COLUMN_NAME']] = row['IS_NULLABLE'] == 'YES'
|
||||
self.column_types.setdefault(row['TABLE_NAME'], {})[row['COLUMN_NAME']] = row['DATA_TYPE']
|
||||
self.tables = sorted(self.columns)
|
||||
self.fks = self.rows('SELECT TABLE_NAME,COLUMN_NAME,REFERENCED_TABLE_NAME,REFERENCED_COLUMN_NAME FROM information_schema.key_column_usage WHERE TABLE_SCHEMA=%s AND REFERENCED_TABLE_NAME IS NOT NULL', (self.name,))
|
||||
self.schema_signature=hashlib.sha256(json.dumps({'columns':self.columns,'fks':sorted(self.fks,key=lambda r:json.dumps(r,sort_keys=True))},sort_keys=True).encode()).hexdigest()
|
||||
self.longest_exclusion=0.0
|
||||
if not {'current_release','data_release','release_member','transform_build','source_upload_draft'}.issubset(self.columns):
|
||||
raise ValueError('Unexpected application schema')
|
||||
|
||||
def rows(self, sql, args=()):
|
||||
with self.conn.cursor() as cursor:
|
||||
cursor.execute(sql, args)
|
||||
return list(cursor.fetchall())
|
||||
|
||||
def execute(self, sql, args=()):
|
||||
with self.conn.cursor() as cursor:
|
||||
return cursor.execute(sql, args)
|
||||
|
||||
def current(self):
|
||||
rows = self.rows("SELECT release_id FROM current_release WHERE dataset_id='quality'")
|
||||
if len(rows) != 1 or not rows[0]['release_id']:
|
||||
raise Busy('Current release not confirmed')
|
||||
fields='status,complete' if 'complete' in self.columns['data_release'] else 'status'
|
||||
release=self.rows('SELECT '+fields+' FROM data_release WHERE id=%s',(rows[0]['release_id'],))
|
||||
if len(release)!=1 or release[0]['status']!='PUBLISHED' or release[0].get('complete',True) not in (True,1):
|
||||
raise Busy('Current release is not a complete publication')
|
||||
return rows[0]['release_id']
|
||||
|
||||
def busy(self):
|
||||
for table in ('task','import_run','data_workflow_operation','analysis_turn','analysis_protocol_execution','quality_inspection_run'):
|
||||
if table not in self.columns or 'status' not in self.columns[table]:
|
||||
continue
|
||||
sql = 'SELECT 1 FROM ' + ident(table) + ' WHERE (status IS NULL OR status NOT IN (' + ','.join(['%s']*len(TERMINAL)) + '))'
|
||||
if table == 'task' and {'next_retry_at','retry_count','max_retries'}.issubset(self.columns[table]):
|
||||
sql += " OR (status='FAILED' AND next_retry_at IS NOT NULL AND retry_count<max_retries)"
|
||||
if self.rows(sql + ' LIMIT 1', TERMINAL):
|
||||
return True
|
||||
if 'analysis_report' in self.columns:
|
||||
# Report rows keep QUEUED after handing work to analysis_turn. The
|
||||
# linked turn, not this stored label, is the execution state.
|
||||
for report in self.rows('SELECT status,turn_id FROM analysis_report'):
|
||||
if report['turn_id']:
|
||||
turns=self.rows('SELECT status FROM analysis_turn WHERE id=%s',(report['turn_id'],))
|
||||
if not turns or turns[0]['status'] not in TERMINAL:return True
|
||||
elif report['status'] not in (*TERMINAL,'BLOCKED'):
|
||||
return True
|
||||
if 'source_candidate' in self.columns:
|
||||
return bool(self.rows("SELECT 1 FROM source_candidate WHERE status IN ('QUEUED','RUNNING','IMPORTING') LIMIT 1"))
|
||||
return False
|
||||
|
||||
@contextmanager
|
||||
def frozen(self, expected):
|
||||
"""Short exclusion against writers; never START TRANSACTION after LOCK TABLES.
|
||||
|
||||
All aliases are intentionally avoided while these MySQL locks are held.
|
||||
Busy jobs and a changed release cause rollback without deleting anything.
|
||||
"""
|
||||
if (self.root/'.runtime/retention/paused').exists():raise Busy('Service maintenance is paused')
|
||||
if self.busy():
|
||||
raise Busy('An application job is active')
|
||||
self.conn.autocommit(False)
|
||||
locked = False
|
||||
acquired=None
|
||||
try:
|
||||
self.execute('LOCK TABLES ' + ','.join(ident(t) + ' WRITE' for t in self.tables))
|
||||
locked = True
|
||||
acquired=time.monotonic()
|
||||
if sorted(next(iter(r.values())) for r in self.rows('SHOW TABLES'))!=self.tables:
|
||||
raise Busy('Application schema changed')
|
||||
if self.current() != expected or self.busy():
|
||||
raise Busy('The publication or job state changed')
|
||||
yield
|
||||
self.conn.commit()
|
||||
except BaseException as error:
|
||||
self.conn.rollback()
|
||||
if isinstance(error,pymysql.err.OperationalError) and error.args[0] in (1205,3572):
|
||||
raise Busy('Application tables are in use') from error
|
||||
raise
|
||||
finally:
|
||||
if locked:
|
||||
self.execute('UNLOCK TABLES')
|
||||
self.longest_exclusion=max(self.longest_exclusion,time.monotonic()-acquired)
|
||||
self.conn.autocommit(True)
|
||||
|
||||
def protected_releases(self, current):
|
||||
releases = self.rows('SELECT id,status,parent_release_id,expected_current_release_id FROM data_release')
|
||||
protected = {current}
|
||||
pending = {r['id'] for r in releases if r['status'] != 'PUBLISHED'}
|
||||
protected.update(pending)
|
||||
# Keep candidate baselines and all their ancestors: review/retry must work.
|
||||
while True:
|
||||
more = {r[c] for r in releases if r['id'] in pending for c in ('parent_release_id','expected_current_release_id') if r[c]}
|
||||
if more.issubset(pending):
|
||||
break
|
||||
pending.update(more); protected.update(more)
|
||||
for fk in self.fks:
|
||||
t, c = fk['TABLE_NAME'], fk['COLUMN_NAME']
|
||||
if fk['REFERENCED_TABLE_NAME'] != 'data_release' or t in MUTABLE_RELEASE_REFS or t in OWNED_RELEASE:
|
||||
continue
|
||||
protected.update(r[c] for r in self.rows('SELECT DISTINCT ' + ident(c) + ' FROM ' + ident(t) + ' WHERE ' + ident(c) + ' IS NOT NULL'))
|
||||
# This field is intentionally not a database FK, but is a real user reference.
|
||||
if 'analysis_session' in self.columns:
|
||||
protected.update(r['release_id'] for r in self.rows('SELECT DISTINCT release_id FROM analysis_session'))
|
||||
if 'source_candidate' in self.columns and 'expected_current_release_id' in self.columns['source_candidate']:
|
||||
protected.update(r['expected_current_release_id'] for r in self.rows("SELECT DISTINCT expected_current_release_id FROM source_candidate WHERE status IN ('DISCOVERED','QUEUED','RUNNING') AND expected_current_release_id IS NOT NULL"))
|
||||
if 'source_change_request' in self.columns:
|
||||
cols = [c for c in ('requested_from_release_id','result_release_id') if c in self.columns['source_change_request']]
|
||||
for row in self.rows('SELECT ' + ','.join(map(ident, cols)) + " FROM source_change_request WHERE status IN ('REQUESTED','APPROVED')") if cols else []:
|
||||
protected.update(v for v in row.values() if v)
|
||||
if 'candidate_reconciliation' in self.columns:
|
||||
for row in self.rows('SELECT release_id,baseline_release_id FROM candidate_reconciliation'):
|
||||
if row['release_id'] in pending and row['baseline_release_id']:
|
||||
protected.add(row['baseline_release_id'])
|
||||
return releases, protected
|
||||
|
||||
def release_step(self, current, state, grace, now):
|
||||
releases, protected = self.protected_releases(current)
|
||||
seen = state.setdefault('releases', {})
|
||||
eligible = []
|
||||
ids = {r['id'] for r in releases}
|
||||
for stale in set(seen) - ids:
|
||||
seen.pop(stale)
|
||||
for row in releases:
|
||||
rid = row['id']
|
||||
if rid in protected:
|
||||
seen.pop(rid, None)
|
||||
elif row['status'] == 'PUBLISHED':
|
||||
seen.setdefault(rid, now)
|
||||
if now - seen[rid] >= grace:
|
||||
eligible.append(rid)
|
||||
if not eligible:
|
||||
return 0
|
||||
rid = sorted(eligible, key=lambda x: seen[x])[0]
|
||||
# Clear optional historical links only. Unknown/business references were
|
||||
# included in protected above and are never cascaded away.
|
||||
for fk in self.fks:
|
||||
t,c = fk['TABLE_NAME'],fk['COLUMN_NAME']
|
||||
if fk['REFERENCED_TABLE_NAME'] == 'data_release' and t in MUTABLE_RELEASE_REFS and self.columns[t][c]:
|
||||
self.execute('UPDATE ' + ident(t) + ' SET ' + ident(c) + '=NULL WHERE ' + ident(c) + '=%s', (rid,))
|
||||
if 'data_workflow_operation' in self.columns:
|
||||
for c in ('release_id','expected_current_release_id'):
|
||||
if c in self.columns['data_workflow_operation']:
|
||||
self.execute('UPDATE data_workflow_operation SET ' + ident(c) + '=NULL WHERE ' + ident(c) + '=%s', (rid,))
|
||||
for t in OWNED_RELEASE:
|
||||
if t in self.columns:
|
||||
self.execute('DELETE FROM ' + ident(t) + ' WHERE release_id=%s', (rid,))
|
||||
self.execute('DELETE FROM data_release WHERE id=%s', (rid,))
|
||||
seen.pop(rid, None)
|
||||
return 1
|
||||
|
||||
def build_step(self, state, grace, now, batch):
|
||||
referenced = {r['transform_build_id'] for r in self.rows('SELECT DISTINCT transform_build_id FROM release_member')}
|
||||
builds = self.rows('SELECT id,status,build_key FROM transform_build')
|
||||
seen = state.setdefault('builds', {})
|
||||
ids = {r['id'] for r in builds}
|
||||
for stale in set(seen) - ids:
|
||||
seen.pop(stale)
|
||||
for row in builds:
|
||||
bid = row['id']
|
||||
unknown_reference = any(self.rows('SELECT 1 FROM ' + ident(f['TABLE_NAME']) + ' WHERE ' + ident(f['COLUMN_NAME']) + '=%s LIMIT 1', (bid,))
|
||||
for f in self.fks if f['REFERENCED_TABLE_NAME'] == 'transform_build' and f['TABLE_NAME'] not in (*BUILD_CHILDREN, 'release_member'))
|
||||
if bid in referenced or unknown_reference or row['status'] not in ('COMPLETED','FAILED','CANCELLED','RETIRED'):
|
||||
seen.pop(bid, None)
|
||||
continue
|
||||
seen.setdefault(bid, now)
|
||||
if row['status'] != 'RETIRED' and now - seen[bid] < grace:
|
||||
continue
|
||||
if row['status'] != 'RETIRED':
|
||||
# Free the unique reuse key before deleting chunks. A new import
|
||||
# can build a fresh result, never reuse this partially emptied one.
|
||||
tombstone = hashlib.sha256(('retired:'+bid+':'+row['build_key']).encode()).hexdigest()
|
||||
self.execute("UPDATE transform_build SET status='RETIRED',build_key=%s WHERE id=%s", (tombstone,bid))
|
||||
for t in BUILD_CHILDREN:
|
||||
if t in self.columns:
|
||||
count = self.execute('DELETE FROM ' + ident(t) + ' WHERE build_id=%s LIMIT ' + str(int(batch)), (bid,))
|
||||
if count:
|
||||
return count, 0
|
||||
self.execute('DELETE FROM transform_build WHERE id=%s', (bid,))
|
||||
seen.pop(bid, None)
|
||||
return 0, 1
|
||||
return 0, 0
|
||||
|
||||
def file_referenced(self, path):
|
||||
for table, cols in self.columns.items():
|
||||
for col in cols:
|
||||
if col.endswith('_path') and self.rows('SELECT 1 FROM ' + ident(table) + ' WHERE ' + ident(col) + '=%s LIMIT 1', (str(path),)):
|
||||
return True
|
||||
return False
|
||||
|
||||
def expired(self, limit=10):
|
||||
return self.rows('SELECT * FROM source_upload_draft WHERE expires_at<=TIMESTAMPADD(MINUTE,-30,CURRENT_TIMESTAMP(6)) ORDER BY expires_at LIMIT ' + str(int(limit)))
|
||||
|
||||
def promotion(self, draft):
|
||||
if not draft.get('candidate_id') or not {'source_revision','source_document','source_candidate'}.issubset(self.columns):
|
||||
return None
|
||||
rows = self.rows("SELECT * FROM source_candidate WHERE id=%s AND status='IMPORTED'", (draft['candidate_id'],))
|
||||
if not rows:
|
||||
return None
|
||||
c = rows[0]
|
||||
docs = {d['id']:d for d in self.rows('SELECT id,source_slot FROM source_document')}
|
||||
for rev in self.rows('SELECT source_document_id,archive_path,sha256,parser_config_hash,file_size FROM source_revision WHERE sha256=%s AND parser_config_hash=%s', (c['sha256'],c['parser_config_hash'])):
|
||||
if docs.get(rev['source_document_id'],{}).get('source_slot') == c['source_slot']:
|
||||
p=Path(rev['archive_path'])
|
||||
if safe_file(p,self.root/'runtimedata/archive/source-revisions') and p.stat().st_size==rev['file_size']:
|
||||
return c,rev,p
|
||||
return None
|
||||
|
||||
def clean_owned_scratch(self, br):
|
||||
import psutil
|
||||
server=self.rows('SELECT @@server_uuid AS value')[0]['value']
|
||||
directory=self.root/'.runtime/retention/scratch'
|
||||
removed=0
|
||||
for path in directory.glob('*.json'):
|
||||
if path.is_symlink():continue
|
||||
value=json.loads(path.read_text());name=value.get('database','')
|
||||
if value.get('kind')!='quality-restore-check' or value.get('server')!=server or not value.get('created') or name==self.name or not re.fullmatch(r'quality_restore_\d{14}_[a-f0-9]{8}',name):continue
|
||||
if time.time()-value['createdAt']<300:continue
|
||||
alive=False
|
||||
for pid_key,created_key in [('ownerPid','ownerCreated'),('childPid','childCreated')]:
|
||||
if pid_key not in value:continue
|
||||
try:
|
||||
process=psutil.Process(value[pid_key])
|
||||
if abs(process.create_time()-value[created_key])<.01 and process.is_running() and process.status()!=psutil.STATUS_ZOMBIE:alive=True
|
||||
except psutil.NoSuchProcess:pass
|
||||
except psutil.Error:alive=True
|
||||
if alive or self.rows('SELECT 1 FROM information_schema.processlist WHERE DB=%s LIMIT 1',(name,)):continue
|
||||
self.execute('DROP DATABASE IF EXISTS '+ident(name));path.unlink();removed+=1
|
||||
return removed
|
||||
|
||||
|
||||
def cleanup_drafts(db, current, state, now, br):
|
||||
removed=0; bytes_removed=0
|
||||
# Checksum expensive files outside the brief database exclusion window.
|
||||
for draft in db.expired():
|
||||
path=Path(draft['source_path']); boundary=db.root/'runtimedata/archive/upload-drafts'
|
||||
info=None
|
||||
if safe_file(path,boundary):
|
||||
info=fingerprint(path)
|
||||
if info[2]!=draft['file_size'] or br.sha(path)!=draft['sha256']:
|
||||
continue
|
||||
promotion=db.promotion(draft)
|
||||
if promotion:
|
||||
c,rev,target=promotion
|
||||
target_stamp=fingerprint(target)
|
||||
if br.sha(target)!=rev['sha256']:
|
||||
continue
|
||||
with db.frozen(current):
|
||||
live=db.rows('SELECT * FROM source_upload_draft WHERE id=%s AND expires_at<=TIMESTAMPADD(MINUTE,-30,CURRENT_TIMESTAMP(6))',(draft['id'],))
|
||||
if not live or live[0]!=draft:
|
||||
continue
|
||||
if info and (not safe_file(path,boundary) or fingerprint(path)!=info):
|
||||
continue
|
||||
if promotion and safe_file(target,db.root/'runtimedata/archive/source-revisions') and fingerprint(target)==target_stamp:
|
||||
# The WHERE identity prevents overwriting a source reconfigured
|
||||
# between preflight hashing and acquiring the database exclusion.
|
||||
db.execute("UPDATE source_candidate SET source_path=%s,archive_path=%s WHERE id=%s AND status='IMPORTED' AND source_path=%s AND sha256=%s AND parser_config_hash=%s",
|
||||
(str(target),str(target),c['id'],c['source_path'],c['sha256'],c['parser_config_hash']))
|
||||
db.execute('DELETE FROM source_upload_draft WHERE id=%s',(draft['id'],))
|
||||
if info:
|
||||
with db.frozen(current):
|
||||
if safe_file(path,boundary) and fingerprint(path)==info and not db.file_referenced(path):
|
||||
bytes_removed+=path.stat().st_size
|
||||
path.unlink()
|
||||
removed+=1
|
||||
# Interrupted uploads can leave no database row at all. Their age and every
|
||||
# direct path reference are rechecked under the same exclusion before unlink.
|
||||
boundary=db.root/'runtimedata/archive/upload-drafts'
|
||||
for path in list(boundary.glob('upload-*.xlsx'))[:100]:
|
||||
if not safe_file(path,boundary) or now-path.stat().st_mtime<86400:
|
||||
continue
|
||||
info=fingerprint(path)
|
||||
with db.frozen(current):
|
||||
if fingerprint(path)==info and not db.file_referenced(path):
|
||||
size=path.stat().st_size;path.unlink();bytes_removed+=size
|
||||
return removed,bytes_removed
|
||||
|
||||
|
||||
def run(root, max_batches=30, grace=1800):
|
||||
root=Path(root).resolve();sys.path.append(str(root/'scripts'))
|
||||
import backup_retention as br
|
||||
import backup
|
||||
import backup_verifier
|
||||
import revision_retention
|
||||
if (root/'.runtime/retention/paused').exists():raise Busy('Service maintenance is paused')
|
||||
readiness=urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||
try:
|
||||
with readiness.open('http://127.0.0.1:5281/actuator/health',timeout=2) as response:
|
||||
if json.load(response).get('status')!='UP':raise Busy('Backend is not ready')
|
||||
except (OSError,ValueError) as error:raise Busy('Backend is not ready') from error
|
||||
state_path=root/'.runtime/retention/state.json'
|
||||
state=json.loads(state_path.read_text()) if state_path.exists() else {}
|
||||
result={'checkedAt':time.time(),'releasesRemoved':0,'buildsRemoved':0,'rowsRemoved':0,'draftsRemoved':0,'bytesRemoved':0}
|
||||
with br.lock(root,timeout=0):
|
||||
result['retiredBackupFoldersRemoved']=br.clean_retired(root)
|
||||
result['privateVerifiersRemoved']=backup_verifier.clean_abandoned(root)
|
||||
db=Database(root)
|
||||
try:
|
||||
policy_path=root/'.runtime/retention/config.json'
|
||||
if policy_path.exists():
|
||||
policy=json.loads(policy_path.read_text());accepted=br.accepted_backend(root)
|
||||
if policy['schemaSignature']!=db.schema_signature or not accepted or policy['acceptedJarSha256']!=accepted[1]:
|
||||
raise Busy('Application version changed; retention compatibility review required')
|
||||
result['scratchRemoved']=db.clean_owned_scratch(br)
|
||||
current=db.current()
|
||||
if db.busy():
|
||||
raise Busy('An application job is active')
|
||||
latest=br.latest(root)
|
||||
# Recover a restore-verified candidate left between verification and
|
||||
# pointer publication by a previous crash, without creating a duplicate.
|
||||
if not latest or latest[1].get('queryEvidence',{}).get('releaseId')!=current:
|
||||
candidates=[]
|
||||
for folder in (root/'output/backups').iterdir() if (root/'output/backups').exists() else []:
|
||||
candidate=br.verified(folder,checksums=False)
|
||||
if candidate and candidate.get('queryEvidence',{}).get('releaseId')==current:candidates.append((candidate.get('createdAt',''),folder))
|
||||
if candidates:
|
||||
selected=max(candidates)[1]
|
||||
if br.verified(selected):br.rotate(root,selected);latest=br.latest(root)
|
||||
if latest:
|
||||
state['lastBackupAt']=max(state.get('lastBackupAt',0),datetime.fromisoformat(latest[1]['createdAt']).timestamp())
|
||||
due=not latest or latest[1].get('queryEvidence',{}).get('releaseId')!=current
|
||||
due=due or time.time()-state.get('lastBackupAt',0)>86400
|
||||
due=due or (state.get('dirtySince') and not state.get('builds') and time.time()-state['dirtySince']>300)
|
||||
if due:
|
||||
database_bytes=int(db.rows('SELECT COALESCE(SUM(DATA_LENGTH+INDEX_LENGTH),0) AS size FROM information_schema.tables WHERE TABLE_SCHEMA=%s',(db.name,))[0]['size'])
|
||||
archive_bytes=sum(p.stat().st_size for p in (root/'runtimedata/archive').rglob('*') if p.is_file())
|
||||
required_free=max(10*1024**3,3*database_bytes+2*archive_bytes+5*1024**3)
|
||||
reserves = [root / '.runtime', root / 'output/backups']
|
||||
for reserve in reserves: reserve.mkdir(parents=True, exist_ok=True)
|
||||
if min(shutil.disk_usage(p).free for p in reserves)<required_free:raise Busy('Insufficient reserve for a verified backup; existing backup preserved')
|
||||
if state.get('lastBackupAttemptRelease')==current and time.time()-state.get('lastBackupAttempt',0)<3600:
|
||||
raise Busy('Waiting before retrying a failed backup')
|
||||
state['lastBackupAttempt']=time.time();state['lastBackupAttemptRelease']=current;br.atomic(state_path,state)
|
||||
logging.info('creating consistent backup; previous verified backup remains available')
|
||||
before=br.backup_ids(root)
|
||||
try:
|
||||
folder=backup.create();backup.verify(folder)
|
||||
except BaseException:
|
||||
br.discard_failed_new(root,before)
|
||||
raise
|
||||
if db.current()!=current:
|
||||
raise Busy('Publication changed while backing up; cleanup deferred')
|
||||
latest=br.latest(root) # rotate is shared with manual/UI backup operations
|
||||
if latest is None or latest[0]!=folder:
|
||||
br.rotate(root,folder);latest=br.latest(root)
|
||||
state['lastBackupAt']=time.time();state.pop('dirtySince',None)
|
||||
br.atomic(state_path,state)
|
||||
logging.info('isolated restore verified; latest backup published')
|
||||
if not latest or latest[1].get('queryEvidence',{}).get('releaseId')!=current:
|
||||
raise Busy('No verified backup for the current publication')
|
||||
# Every data-deletion batch rechecks both job state and current pointer.
|
||||
deadline=time.monotonic()+20
|
||||
for _ in range(max_batches):
|
||||
with db.frozen(current):
|
||||
releases=db.release_step(current,state,grace,time.time())
|
||||
rows,builds=db.build_step(state,grace,time.time(),1000)
|
||||
result['releasesRemoved']+=releases;result['rowsRemoved']+=rows;result['buildsRemoved']+=builds
|
||||
if not (releases or rows or builds) or time.monotonic()>deadline or db.longest_exclusion>.5:
|
||||
break
|
||||
time.sleep(.05)
|
||||
drafts,size=cleanup_drafts(db,current,state,time.time(),br)
|
||||
result['draftsRemoved']=drafts;result['bytesRemoved']=size
|
||||
sources=revision_retention.Cleaner(db,br,safe_file,fingerprint,Busy)
|
||||
result.update(sources.collect(current,state,time.time(),grace))
|
||||
result['maxExclusiveSeconds']=round(db.longest_exclusion,4)
|
||||
if any(result[k] for k in ('releasesRemoved','rowsRemoved','buildsRemoved','draftsRemoved','bytesRemoved','sourceRevisionsRemoved','sourceArchiveBytesRemoved')):
|
||||
state.setdefault('dirtySince',time.time())
|
||||
state['lastResult']=result;state.pop('lastError',None)
|
||||
state['consecutiveFailures']=0
|
||||
br.atomic(state_path,state)
|
||||
return result
|
||||
except Busy as error:
|
||||
state['lastDeferred']={'time':time.time(),'reason':str(error)}
|
||||
state['lastResult']=result;br.atomic(state_path,state)
|
||||
raise
|
||||
finally:
|
||||
db.conn.close()
|
||||
|
||||
|
||||
def install(root):
|
||||
root=Path(root).resolve();sys.path.insert(0,str(root/'scripts'));import backup_retention as br
|
||||
python=root/'.runtime/retention-env/bin/python3'
|
||||
if not python.is_file():
|
||||
raise ValueError('Private maintenance runtime missing')
|
||||
files=[Path(__file__).resolve(),root/'scripts/backup_retention.py',root/'scripts/revision_retention.py']
|
||||
db=Database(root)
|
||||
try:
|
||||
accepted=br.accepted_backend(root)
|
||||
if not accepted:raise ValueError('Accepted backend runtime is not registered')
|
||||
br.atomic(root/'.runtime/retention/config.json',{'schemaSignature':db.schema_signature,
|
||||
'acceptedJarSha256':accepted[1],'retirementGraceSeconds':1800,'intervalSeconds':300,
|
||||
'backupIntervalSeconds':86400,'draftExpiryGraceSeconds':1800,'sourceRevisionGraceSeconds':1800})
|
||||
finally:db.conn.close()
|
||||
digest=hashlib.sha256(b''.join(p.read_bytes() for p in files)).hexdigest()
|
||||
version=SUPPORT/'retention'/digest;version.mkdir(parents=True,exist_ok=True,mode=0o700)
|
||||
for p in files:
|
||||
shutil.copy2(p,version/p.name);(version/p.name).chmod(0o600)
|
||||
legacy=root/'.runtime/current-snapshot-20261109'
|
||||
if legacy.is_dir() and not legacy.is_symlink() and (legacy/'manifest.json').is_file():
|
||||
br.atomic(root/'.runtime/retention/legacy.json',{'path':'.runtime/current-snapshot-20261109','manifestSha256':br.sha(legacy/'manifest.json')})
|
||||
logs=SUPPORT/'logs';logs.mkdir(parents=True,exist_ok=True,mode=0o700)
|
||||
plist=Path.home()/'Library/LaunchAgents'/(LABEL+'.plist')
|
||||
value={'Label':LABEL,'ProgramArguments':[str(python),str(version/'retention.py'),'run','--root',str(root)],
|
||||
'WorkingDirectory':str(root),'RunAtLoad':True,'StartInterval':300,'ProcessType':'Background','Nice':10,
|
||||
'LowPriorityIO':True,'ExitTimeOut':300,'EnvironmentVariables':{'PYTHONDONTWRITEBYTECODE':'1'},
|
||||
'StandardOutPath':str(logs/'retention-launchd.log'),'StandardErrorPath':str(logs/'retention-launchd-error.log')}
|
||||
subprocess.run(['/bin/launchctl','bootout','gui/'+str(os.getuid())+'/'+LABEL],capture_output=True)
|
||||
with plist.open('wb') as out:plistlib.dump(value,out)
|
||||
plist.chmod(0o600)
|
||||
subprocess.run(['/bin/launchctl','enable','gui/'+str(os.getuid())+'/'+LABEL],check=True)
|
||||
for attempt in range(50):
|
||||
loaded=subprocess.run(['/bin/launchctl','bootstrap','gui/'+str(os.getuid()),str(plist)],capture_output=True)
|
||||
if loaded.returncode==0:break
|
||||
time.sleep(.2)
|
||||
else:raise RuntimeError('Maintenance registration did not complete')
|
||||
print('Installed five-minute idle maintenance and daily/publication backup checks')
|
||||
|
||||
|
||||
def main():
|
||||
parser=argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action',choices=['run','install','status'])
|
||||
parser.add_argument('--root',required=True)
|
||||
args=parser.parse_args();root=Path(args.root).resolve();sys.path.append(str(root/'scripts'));import backup_retention as br
|
||||
if args.action=='install':
|
||||
with br.lock(root,timeout=0):install(root)
|
||||
return
|
||||
if args.action=='status':
|
||||
p=root/'.runtime/retention/state.json';print(p.read_text() if p.exists() else '{}');return
|
||||
logs=SUPPORT/'logs';logs.mkdir(parents=True,exist_ok=True,mode=0o700)
|
||||
handler=RotatingFileHandler(logs/'retention.log',maxBytes=2*1024*1024,backupCount=2)
|
||||
handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %(message)s'))
|
||||
logging.getLogger().addHandler(handler);logging.getLogger().setLevel(logging.INFO)
|
||||
try:
|
||||
result=run(root);logging.info('maintenance complete %s',json.dumps(result))
|
||||
except (Busy,br.Busy) as error:
|
||||
logging.info('maintenance deferred: %s',str(error))
|
||||
except Exception as error:
|
||||
p=root/'.runtime/retention/state.json';state=json.loads(p.read_text()) if p.exists() else {}
|
||||
state['lastError']={'time':time.time(),'type':type(error).__name__};br.atomic(p,state)
|
||||
state['consecutiveFailures']=min(1000,state.get('consecutiveFailures',0)+1);br.atomic(p,state)
|
||||
logging.error('maintenance failed safely: %s',type(error).__name__)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,351 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Pinned macOS service guardian. Never initializes or imports a database."""
|
||||
import argparse
|
||||
import fcntl
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
from logging.handlers import RotatingFileHandler
|
||||
import os
|
||||
from pathlib import Path
|
||||
import plistlib
|
||||
import shutil
|
||||
import signal
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
import psutil
|
||||
|
||||
LABEL = 'com.lyunlab.quality-center-service'
|
||||
HOME = Path.home() / 'Library/Application Support/QualityCenterService'
|
||||
CONFIG = HOME / 'service.json'
|
||||
PLIST = Path.home() / 'Library/LaunchAgents' / (LABEL + '.plist')
|
||||
DOMAIN = 'gui/' + str(os.getuid())
|
||||
|
||||
|
||||
def atomic(path, value):
|
||||
path = Path(path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
temporary = path.with_name(path.name + '.tmp-' + str(os.getpid()))
|
||||
with temporary.open('w') as stream:
|
||||
os.chmod(temporary, 0o600)
|
||||
json.dump(value, stream, ensure_ascii=False, indent=2)
|
||||
temporary.replace(path)
|
||||
|
||||
|
||||
def stamp(pid):
|
||||
return subprocess.run(['/bin/ps', '-p', str(pid), '-o', 'lstart='],
|
||||
capture_output=True, text=True).stdout.strip()
|
||||
|
||||
|
||||
def matches(record, command=None):
|
||||
"""True=confirmed identity, False=gone/reused, None=unknown identity."""
|
||||
if not record:
|
||||
return False
|
||||
try:
|
||||
p = psutil.Process(record['pid'])
|
||||
if not p.is_running() or p.status() == psutil.STATUS_ZOMBIE:
|
||||
return False
|
||||
if 'created' in record and abs(p.create_time() - record['created']) > .01:
|
||||
return False
|
||||
if 'started' in record and stamp(p.pid) != record['started']:
|
||||
return False
|
||||
expected = command or record['command']
|
||||
if p.uids().real != os.getuid():
|
||||
return None
|
||||
if Path(p.exe()).resolve() != Path(expected[0]).resolve() or p.cmdline() != expected:
|
||||
return None
|
||||
return True
|
||||
except psutil.NoSuchProcess:
|
||||
return False
|
||||
except (psutil.Error, KeyError, ValueError, OSError):
|
||||
return None
|
||||
|
||||
|
||||
def record(pid, command):
|
||||
return {'pid': pid, 'created': psutil.Process(pid).create_time(),
|
||||
'command': command, 'started': stamp(pid)}
|
||||
|
||||
|
||||
class Guardian:
|
||||
def __init__(self, config, home=HOME):
|
||||
self.config = config
|
||||
self.home = Path(home)
|
||||
self.root = Path(config['root']).resolve()
|
||||
self.runtime = self.root / '.runtime'
|
||||
self.records = {}
|
||||
self.children = {}
|
||||
self.next_start = {'mysql': 0, 'backend': 0}
|
||||
self.restarts = {'mysql': 0, 'backend': 0}
|
||||
self.stopping = False
|
||||
self.opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||
|
||||
def preflight(self):
|
||||
self.value = json.loads((self.runtime / 'local-config.json').read_text())
|
||||
if not self.value.get('databaseReady') or not (self.runtime / 'mysql-data/auto.cnf').is_file():
|
||||
raise RuntimeError('Existing initialized database not confirmed; refusing bootstrap')
|
||||
self.jar = self.runtime / 'backend-builds' / (self.config['jarSha256'] + '.jar')
|
||||
if hashlib.sha256(self.jar.read_bytes()).hexdigest() != self.config['jarSha256']:
|
||||
raise RuntimeError('Accepted backend digest mismatch')
|
||||
for binary in [self.runtime / 'mysql/bin/mysqld', self.runtime / 'jdk/Contents/Home/bin/java']:
|
||||
if not os.access(binary, os.X_OK):
|
||||
raise RuntimeError('Required runtime missing')
|
||||
|
||||
def command(self, role):
|
||||
r = self.runtime
|
||||
if role == 'backend':
|
||||
return [str(r / 'jdk/Contents/Home/bin/java'), '-Xms128m', '-Xmx2048m', '-jar', str(self.jar)]
|
||||
return [str(r / 'mysql/bin/mysqld'), '--no-defaults', '--basedir=' + str(r / 'mysql'),
|
||||
'--datadir=' + str(r / 'mysql-data'), '--port=' + str(self.value['mysqlPort']),
|
||||
'--bind-address=127.0.0.1', '--socket=' + str(r / 'mysql.sock'),
|
||||
'--pid-file=' + str(r / 'mysql-server.pid'),
|
||||
'--log-error=' + str(self.root / 'output/logs/mysql-server.log'), '--mysqlx=OFF',
|
||||
'--lower-case-table-names=2', '--max-connections=60', '--innodb-buffer-pool-size=256M',
|
||||
*(['--skip-log-bin'] if self.value.get('binaryLogEnabled', True) is False else [])]
|
||||
|
||||
def environment(self):
|
||||
v, r = self.value, self.runtime
|
||||
env = os.environ.copy()
|
||||
env.update({
|
||||
'JAVA_HOME': str(r / 'jdk/Contents/Home'),
|
||||
'PATH': str(r / 'jdk/Contents/Home/bin') + ':' + self.config['path'],
|
||||
'MYSQL_URL': f'jdbc:mysql://127.0.0.1:{v["mysqlPort"]}/{v["mysqlDatabase"]}?useUnicode=true&characterEncoding=utf8&serverTimezone=UTC&forceConnectionTimeZoneToSession=true&useSSL=false&allowPublicKeyRetrieval=true',
|
||||
'MYSQL_USER': v['mysqlUser'], 'MYSQL_PASSWORD': v['mysqlPassword'],
|
||||
'MYSQL_LEGACY_TIMESTAMP_OFFSET_SECONDS': str(v.get('legacyTimestampOffsetSeconds', 'UNCONFIRMED')),
|
||||
'QUALITY_BOOTSTRAP_ADMIN_USERNAME': v['adminUsername'],
|
||||
'QUALITY_BOOTSTRAP_ADMIN_PASSWORD': v['adminPassword'],
|
||||
'QUALITY_ARCHIVE_ROOT': str(self.root / 'runtimedata/archive'),
|
||||
'QUALITY_ALLOWED_ROOTS': str(self.root / '质量数据/系统原始数据示例'),
|
||||
'QUALITY_DATA_WORKFLOW_TEST_MODE': 'true',
|
||||
'QUALITY_ANALYSIS_PROTOCOL_V1_ENABLED': str(v.get('analysisProtocolV1Enabled', False)).lower(),
|
||||
'QUALITY_ANALYSIS_AUTONOMOUS_ENABLED': str(v.get('analysisAutonomousEnabled', False)).lower(),
|
||||
'QUALITY_PROJECT_ROOT': str(self.root), 'QUALITY_DEFAULT_SOURCE_YEAR': '2026',
|
||||
'SERVER_ADDRESS': '127.0.0.1', 'PORT': '5281', 'PYTHONDONTWRITEBYTECODE': '1',
|
||||
})
|
||||
log_config = r / 'operations/logback-quality.xml'
|
||||
if log_config.is_file() and not log_config.is_symlink():
|
||||
env['LOGGING_CONFIG'] = str(log_config)
|
||||
env['QUALITY_LOG_DIR'] = str(self.root / 'output/logs')
|
||||
return env
|
||||
|
||||
def port(self, role):
|
||||
return self.value['mysqlPort'] if role == 'mysql' else 5281
|
||||
|
||||
def occupied(self, role):
|
||||
try:
|
||||
with socket.create_connection(('127.0.0.1', self.port(role)), timeout=.5):
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
def db_ready(self):
|
||||
result = subprocess.run([str(self.runtime / 'mysql/bin/mysql'),
|
||||
'--defaults-file=' + str(self.runtime / 'mysql-root.cnf'), '--connect-timeout=2',
|
||||
'--batch', '--skip-column-names', '-e', 'SELECT 1;'],
|
||||
stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=4)
|
||||
return result.returncode == 0 and result.stdout.strip() == b'1'
|
||||
|
||||
def backend_ready(self):
|
||||
try:
|
||||
with self.opener.open('http://127.0.0.1:5281/actuator/health', timeout=2) as response:
|
||||
return json.load(response).get('status') == 'UP'
|
||||
except (OSError, ValueError):
|
||||
return False
|
||||
|
||||
def adopt(self, role):
|
||||
path = self.runtime / (role + '.process.json')
|
||||
if not path.exists():
|
||||
return False
|
||||
info = json.loads(path.read_text())
|
||||
identity = matches(info, self.command(role))
|
||||
if identity is None:
|
||||
raise RuntimeError(role + ': existing process identity cannot be confirmed')
|
||||
if identity:
|
||||
self.records[role] = record(info['pid'], self.command(role))
|
||||
logging.info('%s adopted pid=%s', role, info['pid'])
|
||||
return bool(identity)
|
||||
|
||||
def ensure(self, role):
|
||||
child = self.children.get(role)
|
||||
if child is not None:
|
||||
child.poll() # Reap exited children before checking PID identity.
|
||||
info = self.records.get(role)
|
||||
state = matches(info, self.command(role))
|
||||
if state is None:
|
||||
raise RuntimeError(role + ': process identity is unknown; no replacement started')
|
||||
if state:
|
||||
return True
|
||||
if info:
|
||||
self.records.pop(role)
|
||||
self.next_start[role] = time.monotonic() + 10
|
||||
logging.warning('%s exited; restart scheduled in 10 seconds', role)
|
||||
if time.monotonic() < self.next_start[role]:
|
||||
return False
|
||||
if self.adopt(role):
|
||||
return True
|
||||
if self.occupied(role):
|
||||
raise RuntimeError(role + ': port occupied by an unconfirmed process')
|
||||
command = self.command(role)
|
||||
logs = self.root / 'output/logs'
|
||||
logs.mkdir(parents=True, exist_ok=True)
|
||||
log_name = ('backend-console.log' if role == 'backend' and
|
||||
(self.runtime / 'operations/logback-quality.xml').is_file() else role + '.log')
|
||||
with (logs / log_name).open('ab') as log:
|
||||
child = subprocess.Popen(command, cwd=self.root, env=self.environment(),
|
||||
stdin=subprocess.DEVNULL, stdout=log, stderr=log, start_new_session=True)
|
||||
self.children[role] = child
|
||||
self.records[role] = info = record(child.pid, command)
|
||||
(self.runtime / (role + '.pid')).write_text(str(child.pid))
|
||||
atomic(self.runtime / (role + '.process.json'), {'pid': child.pid, 'started': info['started']})
|
||||
self.restarts[role] += 1
|
||||
logging.info('%s started pid=%s', role, child.pid)
|
||||
return True
|
||||
|
||||
def tick(self):
|
||||
mysql_alive = self.ensure('mysql')
|
||||
ready = mysql_alive and self.db_ready()
|
||||
if ready:
|
||||
self.ensure('backend')
|
||||
atomic(self.home / 'state.json', {'updatedAt': time.time(), 'supervisorPid': os.getpid(),
|
||||
'processes': self.records, 'starts': self.restarts,
|
||||
'databaseReady': bool(ready), 'backendReady': self.backend_ready()})
|
||||
|
||||
def run(self):
|
||||
self.preflight()
|
||||
with (self.runtime / 'quality-service.lock').open('a') as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
for sig in (signal.SIGTERM, signal.SIGINT):
|
||||
signal.signal(sig, lambda *_: setattr(self, 'stopping', True))
|
||||
logging.info('guardian started; accepted backend=%s', self.config['jarSha256'])
|
||||
while not self.stopping:
|
||||
try:
|
||||
self.tick()
|
||||
except Exception as error:
|
||||
# Never log exception arguments: database credentials may be embedded.
|
||||
detail = str(error) if type(error) is RuntimeError else type(error).__name__
|
||||
logging.error('check failed: %s; leaving existing processes intact', detail)
|
||||
atomic(self.home / 'error.json', {'time': time.time(), 'type': type(error).__name__})
|
||||
for _ in range(30):
|
||||
if self.stopping:
|
||||
break
|
||||
time.sleep(.1)
|
||||
logging.info('guardian stopped; service processes remain available for maintenance')
|
||||
|
||||
|
||||
def ctl(*args, check=True):
|
||||
return subprocess.run(['/bin/launchctl', *args], check=check, capture_output=True, text=True)
|
||||
|
||||
|
||||
def install(root, jar, clear_maintenance_pause=True):
|
||||
root, jar = Path(root).resolve(), Path(jar).resolve()
|
||||
cfg = {'root': str(root), 'jarSha256': hashlib.sha256(jar.read_bytes()).hexdigest(),
|
||||
'path': '/opt/homebrew/bin:/usr/local/bin:/Library/Frameworks/Python.framework/Versions/3.13/bin:/usr/bin:/bin:/usr/sbin:/sbin'}
|
||||
Guardian(cfg).preflight()
|
||||
HOME.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
HOME.chmod(0o700)
|
||||
script = Path(__file__).resolve()
|
||||
digest = hashlib.sha256(script.read_bytes()).hexdigest()
|
||||
target = HOME / 'versions' / digest / 'service.py'
|
||||
target.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
shutil.copyfile(script, target)
|
||||
target.chmod(0o600)
|
||||
# Remove only the previous guardian; adopted MySQL/backend processes keep running.
|
||||
ctl('bootout', DOMAIN + '/' + LABEL, check=False)
|
||||
atomic(CONFIG, cfg)
|
||||
logs = HOME / 'logs'
|
||||
logs.mkdir(mode=0o700, exist_ok=True)
|
||||
plist = {'Label': LABEL, 'ProgramArguments': [str(Path(sys.executable).resolve()), str(target), 'run'],
|
||||
'WorkingDirectory': str(HOME), 'RunAtLoad': True, 'KeepAlive': True,
|
||||
'ThrottleInterval': 10, 'ProcessType': 'Background', 'ExitTimeOut': 30,
|
||||
'AbandonProcessGroup': True,
|
||||
'EnvironmentVariables': {'PYTHONDONTWRITEBYTECODE': '1'},
|
||||
'StandardOutPath': str(logs / 'launchd.log'), 'StandardErrorPath': str(logs / 'launchd-error.log')}
|
||||
PLIST.parent.mkdir(exist_ok=True)
|
||||
with PLIST.open('wb') as stream:
|
||||
plistlib.dump(plist, stream)
|
||||
PLIST.chmod(0o600)
|
||||
ctl('enable', DOMAIN + '/' + LABEL)
|
||||
for attempt in range(50):
|
||||
loaded=ctl('bootstrap', DOMAIN, str(PLIST), check=False)
|
||||
if loaded.returncode==0:break
|
||||
time.sleep(.2)
|
||||
else:raise RuntimeError('Service registration did not complete; existing processes preserved')
|
||||
if clear_maintenance_pause:
|
||||
(root/'.runtime/retention/paused').unlink(missing_ok=True)
|
||||
print('Installed login startup and continuous guardian:', PLIST)
|
||||
|
||||
|
||||
def status():
|
||||
loaded = ctl('print', DOMAIN + '/' + LABEL, check=False)
|
||||
state_path = HOME / 'state.json'
|
||||
state = json.loads(state_path.read_text()) if state_path.exists() else {}
|
||||
output = {'launchdLoaded': loaded.returncode == 0, 'stateFresh': time.time() - state.get('updatedAt', 0) < 20,
|
||||
'databaseReady': state.get('databaseReady'), 'backendReady': state.get('backendReady'),
|
||||
'supervisorPid': state.get('supervisorPid'), 'starts': state.get('starts')}
|
||||
output['processes'] = {k: {'pid': v['pid'], 'confirmed': matches(v)} for k, v in state.get('processes', {}).items()}
|
||||
print(json.dumps(output, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
def stop_services():
|
||||
cfg = json.loads(CONFIG.read_text())
|
||||
guardian = Guardian(cfg)
|
||||
guardian.preflight()
|
||||
for role in ('backend', 'mysql'):
|
||||
if not guardian.adopt(role):
|
||||
continue
|
||||
info = guardian.records[role]
|
||||
if matches(info) is not True:
|
||||
raise RuntimeError('Cannot confirm process identity')
|
||||
psutil.Process(info['pid']).terminate()
|
||||
deadline = time.monotonic() + 300
|
||||
while matches(info) is True and time.monotonic() < deadline:
|
||||
time.sleep(.2)
|
||||
if matches(info) is not False:
|
||||
raise RuntimeError('Graceful shutdown incomplete; no forced termination performed')
|
||||
print(role + ' stopped gracefully')
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action', choices=['install', 'run', 'status', 'start', 'pause', 'stop'])
|
||||
parser.add_argument('--root')
|
||||
parser.add_argument('--jar')
|
||||
args = parser.parse_args()
|
||||
if args.action == 'install':
|
||||
if not args.root or not args.jar:
|
||||
parser.error('install requires --root and --jar')
|
||||
install(args.root, args.jar)
|
||||
elif args.action == 'status':
|
||||
status()
|
||||
elif args.action in ('pause', 'stop'):
|
||||
cfg = json.loads(CONFIG.read_text())
|
||||
atomic(Path(cfg['root'])/'.runtime/retention/paused', {'since':time.time()})
|
||||
ctl('disable', DOMAIN + '/' + LABEL)
|
||||
ctl('bootout', DOMAIN + '/' + LABEL, check=False)
|
||||
if args.action == 'stop':
|
||||
stop_services()
|
||||
print('Automatic restart disabled until the start command is used')
|
||||
elif args.action == 'start':
|
||||
cfg = json.loads(CONFIG.read_text())
|
||||
(Path(cfg['root'])/'.runtime/retention/paused').unlink(missing_ok=True)
|
||||
ctl('enable', DOMAIN + '/' + LABEL)
|
||||
if ctl('print', DOMAIN + '/' + LABEL, check=False).returncode:
|
||||
ctl('bootstrap', DOMAIN, str(PLIST))
|
||||
else:
|
||||
ctl('kickstart', DOMAIN + '/' + LABEL)
|
||||
status()
|
||||
else:
|
||||
logs = HOME / 'logs'
|
||||
logs.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
handler = RotatingFileHandler(logs / 'guardian.log', maxBytes=5*1024*1024, backupCount=4)
|
||||
handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %(message)s'))
|
||||
logging.getLogger().addHandler(handler)
|
||||
logging.getLogger().setLevel(logging.INFO)
|
||||
Guardian(json.loads(CONFIG.read_text())).run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Verify the actual JDK, accepted logging libraries, JDBC and cross-language registry."""
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import uuid
|
||||
import zipfile
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0,str(ROOT/'scripts'))
|
||||
import backup
|
||||
import backup_retention as br
|
||||
import managed_test_resources as resources
|
||||
|
||||
|
||||
HARNESS=r'''
|
||||
import java.sql.*;
|
||||
import javax.sql.DataSource;
|
||||
import cn.yunsheng.quality.testsupport.OwnedTestDatabase;
|
||||
public class TestDbProbe {
|
||||
static class DS implements DataSource {
|
||||
public Connection getConnection() throws SQLException {return DriverManager.getConnection(System.getenv("TEST_DB_URL"),"root",System.getenv("TEST_DB_PASSWORD"));}
|
||||
public Connection getConnection(String u,String p)throws SQLException{return DriverManager.getConnection(System.getenv("TEST_DB_URL"),u,p);}
|
||||
public java.io.PrintWriter getLogWriter(){return null;} public void setLogWriter(java.io.PrintWriter w){}
|
||||
public void setLoginTimeout(int t){} public int getLoginTimeout(){return 5;}
|
||||
public java.util.logging.Logger getParentLogger(){return java.util.logging.Logger.getGlobal();}
|
||||
public <T>T unwrap(Class<T> c)throws SQLException{throw new SQLException();} public boolean isWrapperFor(Class<?> c){return false;}
|
||||
}
|
||||
public static void main(String[] a)throws Exception{
|
||||
Class.forName("com.mysql.cj.jdbc.Driver");DS ds=new DS();String name=a[1];
|
||||
if(a[0].equals("normal")){OwnedTestDatabase.create(ds,name);OwnedTestDatabase.drop(ds,name);}
|
||||
else if(a[0].equals("empty")){
|
||||
OwnedTestDatabase.createEmpty(ds,name);
|
||||
try(Connection c=ds.getConnection();Statement s=c.createStatement();ResultSet rs=s.executeQuery("SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='"+name+"'")){
|
||||
rs.next();if(rs.getInt(1)!=0)throw new AssertionError("Fresh schema is not empty");}
|
||||
OwnedTestDatabase.seal(ds,name);OwnedTestDatabase.drop(ds,name);
|
||||
}else OwnedTestDatabase.create(ds,name);
|
||||
System.out.println("PROBE_OK");
|
||||
}
|
||||
}
|
||||
'''
|
||||
|
||||
LOG_PROBE=r'''
|
||||
import ch.qos.logback.classic.LoggerContext;
|
||||
import ch.qos.logback.classic.joran.JoranConfigurator;
|
||||
import ch.qos.logback.core.status.Status;
|
||||
import java.nio.file.*;
|
||||
public class LogProbe {
|
||||
public static void main(String[] args)throws Exception{
|
||||
LoggerContext c=new LoggerContext();c.setMDCAdapter(new ch.qos.logback.classic.util.LogbackMDCAdapter());c.putProperty("QUALITY_LOG_DIR",args[1]);
|
||||
JoranConfigurator config=new JoranConfigurator();config.setContext(c);config.doConfigure(args[0]);
|
||||
for(Status s:c.getStatusManager().getCopyOfStatusList())if(s.getLevel()>=Status.ERROR)throw new AssertionError(s.toString());
|
||||
var logger=c.getLogger("probe");
|
||||
for(int i=0;i<12000;i++)logger.info("{} {}",java.util.UUID.randomUUID(),java.util.UUID.randomUUID());
|
||||
Thread.sleep(1500);
|
||||
for(Status s:c.getStatusManager().getCopyOfStatusList())if(s.getLevel()>=Status.ERROR)throw new AssertionError(s.toString());
|
||||
c.stop();System.out.println("LOG_PROBE_OK");
|
||||
}
|
||||
}
|
||||
'''
|
||||
|
||||
|
||||
class NativeOperationsTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
if not (ROOT / '.runtime/local-config.json').is_file(): raise unittest.SkipTest('Accepted local JDK/MySQL runtime is not configured')
|
||||
cls.temp=tempfile.TemporaryDirectory();cls.build=Path(cls.temp.name).resolve();cls.libs=cls.build/'lib';cls.libs.mkdir()
|
||||
cfg=json.loads((Path.home()/'Library/Application Support/QualityCenterService/service.json').read_text())
|
||||
jar=ROOT/'.runtime/backend-builds'/(cfg['jarSha256']+'.jar')
|
||||
with zipfile.ZipFile(jar) as z:
|
||||
for n in z.namelist():
|
||||
if n.startswith('BOOT-INF/lib/') and any(x in Path(n).name for x in ['logback-','slf4j-api-','spring-boot-4.','spring-core-','mysql-connector-']):
|
||||
(cls.libs/Path(n).name).write_bytes(z.read(n))
|
||||
(cls.build/'TestDbProbe.java').write_text(HARNESS);(cls.build/'LogProbe.java').write_text(LOG_PROBE)
|
||||
cls.java=ROOT/'.runtime/jdk/Contents/Home/bin/java';cls.javac=cls.java.with_name('javac')
|
||||
cls.classes=cls.build/'classes';cls.classes.mkdir()
|
||||
helper=ROOT/'backend/src/test/java/cn/yunsheng/quality/testsupport/OwnedTestDatabase.java'
|
||||
p=subprocess.run([str(cls.javac),'-cp',str(cls.libs/'*'),'-d',str(cls.classes),str(helper),str(cls.build/'TestDbProbe.java'),str(cls.build/'LogProbe.java')],text=True,capture_output=True)
|
||||
if p.returncode:raise RuntimeError(p.stderr)
|
||||
@classmethod
|
||||
def tearDownClass(cls):cls.temp.cleanup()
|
||||
|
||||
def test_java_registration_normal_drop_fresh_schema_and_expired_orphan_cleanup(self):
|
||||
cfg=json.loads((ROOT/'.runtime/local-config.json').read_text())
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root=Path(directory).resolve();br.atomic(root/'.runtime/local-config.json',{'mysqlDatabase':cfg['mysqlDatabase']})
|
||||
env=os.environ.copy();env.update({'TEST_DB_URL':f"jdbc:mysql://127.0.0.1:{cfg['mysqlPort']}/?useSSL=false&allowPublicKeyRetrieval=true",'TEST_DB_PASSWORD':cfg['rootPassword']})
|
||||
names=[]
|
||||
try:
|
||||
for mode in ['normal','empty','orphan']:
|
||||
name='quality_sales_type_test_'+uuid.uuid4().hex;names.append(name)
|
||||
p=subprocess.run([str(self.java),'-cp',str(self.classes)+os.pathsep+str(self.libs/'*'),'TestDbProbe',mode,name],cwd=root,env=env,text=True,capture_output=True,timeout=30)
|
||||
self.assertEqual(p.returncode,0,p.stderr);self.assertIn('PROBE_OK',p.stdout)
|
||||
if mode!='orphan':self.assertIsNone(resources.load(root,name))
|
||||
else:
|
||||
record=resources.load(root,name);self.assertTrue(record['ready'])
|
||||
self.assertFalse(resources.active_process(record['ownerPid'],record['ownerCreated']))
|
||||
result=resources.cleanup(root,backup.client,record['expiresAt']+1)
|
||||
self.assertEqual(result['testDatabasesRemoved'],1);self.assertIsNone(resources.load(root,name))
|
||||
finally:
|
||||
for name in names:backup.client('DROP DATABASE IF EXISTS `'+name+'`;')
|
||||
|
||||
def test_actual_logback_configuration_rolls_and_bounds_files(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root=Path(directory);xml=(ROOT/'deploy/macos/logback-quality.xml').read_text()
|
||||
# Exercise identical policy at a smaller scale, retaining asynchronous-roll headroom.
|
||||
self.assertIn('<maxFileSize>10MB</maxFileSize>',xml);self.assertIn('<totalSizeCap>80MB</totalSizeCap>',xml)
|
||||
config=root/'logback.xml';config.write_text(xml.replace('10MB','10KB').replace('80MB','80KB'))
|
||||
p=subprocess.run([str(self.java),'-cp',str(self.classes)+os.pathsep+str(self.libs/'*'),'LogProbe',str(config),str(root)],text=True,capture_output=True,timeout=30)
|
||||
self.assertEqual(p.returncode,0,p.stderr+p.stdout);self.assertIn('LOG_PROBE_OK',p.stdout)
|
||||
archives=list(root.glob('backend.*.log.gz'));self.assertTrue(archives)
|
||||
self.assertLessEqual(sum(f.stat().st_size for f in archives),90*1024)
|
||||
self.assertLessEqual(sum(f.stat().st_size for f in archives)+(root/'backend.log').stat().st_size,100*1024)
|
||||
self.assertLessEqual((root/'backend.log').stat().st_size,12*1024)
|
||||
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
@@ -0,0 +1,195 @@
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import uuid
|
||||
from unittest.mock import patch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / 'scripts'))
|
||||
import backup
|
||||
import backup_retention as br
|
||||
import log_retention as logs
|
||||
import managed_test_resources as resources
|
||||
spec = importlib.util.spec_from_file_location('quality_operations', Path(__file__).with_name('operations.py'))
|
||||
ops = importlib.util.module_from_spec(spec); spec.loader.exec_module(ops)
|
||||
|
||||
|
||||
class OperationsTests(unittest.TestCase):
|
||||
def sample(self, **extra):
|
||||
return {'at': 200000., 'coreBytes': 12*ops.GIB, 'permanentBytes': 40*ops.GIB,
|
||||
'freeBytes': 500*ops.GIB, 'overdueTestResources': 0, **extra}
|
||||
|
||||
def test_capacity_levels_free_space_failures_and_growth_are_separate(self):
|
||||
self.assertEqual(ops.conditions(self.sample(), ops.DEFAULTS, {}, []), {})
|
||||
self.assertEqual(ops.conditions(self.sample(coreBytes=30*ops.GIB), ops.DEFAULTS, {}, [])['capacity']['severity'], 'warning')
|
||||
self.assertIn('temporary',ops.conditions(self.sample(temporaryBytes=30*ops.GIB),ops.DEFAULTS,{},[]))
|
||||
self.assertIn('artifacts',ops.conditions(self.sample(artifactBytes=50*ops.GIB),ops.DEFAULTS,{},[]))
|
||||
found = ops.conditions(self.sample(coreBytes=50*ops.GIB, freeBytes=19*ops.GIB, overdueTestResources=1),
|
||||
ops.DEFAULTS, {'consecutiveFailures': 3}, [])
|
||||
self.assertEqual(set(found), {'capacity', 'disk', 'maintenance', 'tests'})
|
||||
old = self.sample(at=200000-86400, permanentBytes=30*ops.GIB)
|
||||
self.assertIn('growth', ops.conditions(self.sample(), ops.DEFAULTS, {}, [old]))
|
||||
self.assertNotIn('maintenance', ops.conditions(self.sample(), ops.DEFAULTS, {'lastDeferred': {'reason': 'active'}}, []))
|
||||
old = self.sample(at=200000-86400)
|
||||
self.assertIn('growth',ops.conditions(self.sample(coreBytes=15*ops.GIB,permanentBytes=43*ops.GIB),ops.DEFAULTS,{},[old]))
|
||||
|
||||
def test_monitor_failure_notifies_after_three_attempts_without_repeated_noise(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
root=Path(d).resolve();sent=[]
|
||||
def notify(*_):sent.append(True);return {'submitted':True}
|
||||
for _ in range(4):ops.failed_run(root,RuntimeError('not logged'),notify)
|
||||
self.assertEqual(len(sent),1)
|
||||
self.assertEqual(json.loads((root/'.runtime/operations/failure.json').read_text())['count'],4)
|
||||
|
||||
def test_alert_deduplication_escalation_recovery_and_failed_delivery_retry(self):
|
||||
sent = []
|
||||
def notify(title, message): sent.append(message); return {'submitted': True}
|
||||
warning = {'capacity': {'severity': 'warning', 'message': 'warning'}}
|
||||
first, events = ops.dispatch(warning, {}, 100, ops.DEFAULTS, notify)
|
||||
second, events = ops.dispatch(warning, first, 200, ops.DEFAULTS, notify)
|
||||
self.assertEqual(len(sent), 1); self.assertFalse(events)
|
||||
critical = {'capacity': {'severity': 'critical', 'message': 'critical'}}
|
||||
third, _ = ops.dispatch(critical, second, 300, ops.DEFAULTS, notify)
|
||||
self.assertEqual(len(sent), 2)
|
||||
_, events = ops.dispatch({}, third, 400, ops.DEFAULTS, notify)
|
||||
self.assertEqual(events[0]['kind'], 'resolved')
|
||||
failed, _ = ops.dispatch(warning, {}, 500, ops.DEFAULTS, lambda *_: {'submitted': False})
|
||||
ops.dispatch(warning, failed, 600, ops.DEFAULTS, notify)
|
||||
self.assertEqual(len(sent), 3)
|
||||
|
||||
def test_registered_and_private_temporary_storage_is_not_permanent_growth(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
root = Path(d).resolve(); support = root / 'support'
|
||||
name = 'quality_retention_test_' + uuid.uuid4().hex[:12]
|
||||
for path in [root/'.runtime/mysql-data/current/data.ibd', root/'.runtime/mysql-data'/name/'sample.ibd',
|
||||
root/'output/backups/current/file.gz', root/'runtimedata/archive/source.xlsx']:
|
||||
path.parent.mkdir(parents=True, exist_ok=True); path.write_bytes(b'x'*8192)
|
||||
br.atomic(resources.registry(root)/(name+'.json'), {'kind':'quality-owned-test-db','root':str(root),
|
||||
'database':name,'token':'1'*32,'created':True,'expiresAt':time.time()+86400})
|
||||
private=root/'.runtime/retention/verify-fixture';private.mkdir(parents=True)
|
||||
(private/'data').write_bytes(b'x'*16384)
|
||||
br.atomic(private/'owner.json',{'kind':'private-quality-restore','directory':str(private)})
|
||||
sample=ops.snapshot(root,support)
|
||||
self.assertGreater(sample['registeredTestBytes'],0); self.assertGreater(sample['privateVerifierBytes'],0)
|
||||
self.assertLess(sample['coreBytes'],sample['mysqlBytes']+sample['backupBytes']+sample['archiveBytes'])
|
||||
|
||||
def test_monitor_still_collects_when_housekeeping_is_deferred_and_history_is_bounded(self):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
root=Path(d).resolve();base=root/'.runtime/operations';base.mkdir(parents=True)
|
||||
br.atomic(base/'policy.json',ops.DEFAULTS)
|
||||
history=[self.sample(at=time.time()-i*100) for i in range(450)]
|
||||
br.atomic(base/'samples.json',history)
|
||||
with patch.object(ops,'snapshot',return_value=self.sample(at=time.time())), \
|
||||
patch.object(ops,'janitor',return_value={'deferred':'Application jobs are active'}):
|
||||
result=ops.run(root,root/'support',lambda *_:{'submitted':True})
|
||||
self.assertIn('sample',result);self.assertIn('deferred',result['cleanup'])
|
||||
self.assertLessEqual(len(json.loads((base/'samples.json').read_text())),330)
|
||||
|
||||
|
||||
class LogRetentionTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp=tempfile.TemporaryDirectory();self.root=Path(self.temp.name).resolve();self.support=self.root/'support'
|
||||
def tearDown(self):self.temp.cleanup()
|
||||
def file(self,name,age=0,size=100):
|
||||
p=self.root/'output/logs'/name;p.parent.mkdir(parents=True,exist_ok=True);p.write_bytes(b'line\n'*(size//5))
|
||||
os.utime(p,(time.time()-age,time.time()-age));return p
|
||||
|
||||
def test_old_closed_logs_removed_but_active_symlinks_and_business_files_preserved(self):
|
||||
old=self.file('old.log',age=8*86400);active=self.file('backend.log',age=8*86400)
|
||||
other=self.file('worker.log',age=8*86400);outside=self.root/'important.xlsx';outside.write_bytes(b'keep')
|
||||
link=old.with_name('link.log');link.symlink_to(outside)
|
||||
result=logs.retain(self.root,self.support,open_paths={str(other)})
|
||||
self.assertEqual(result['logsRemoved'],1);self.assertFalse(old.exists())
|
||||
self.assertTrue(active.exists());self.assertTrue(other.exists());self.assertTrue(outside.exists());self.assertTrue(link.is_symlink())
|
||||
|
||||
def test_per_class_cap_prunes_old_archives_and_bounds_a_closed_console_tail(self):
|
||||
old=self.file('worker.log.1',age=100,size=100);new=self.file('worker.log',age=0,size=100)
|
||||
logs.retain(self.root,self.support,open_paths=set(),max_bytes=150)
|
||||
self.assertFalse(old.exists());self.assertTrue(new.exists())
|
||||
big=self.file('huge.log',size=1000)
|
||||
logs.retain(self.root,self.support,open_paths=set(),max_bytes=100)
|
||||
self.assertLessEqual(big.stat().st_size,100)
|
||||
|
||||
def test_mysql_rotation_uses_reopen_and_recovers_after_interruption(self):
|
||||
source=self.file('mysql-server.log',age=86400,size=100);original=source.read_bytes()
|
||||
with self.assertRaises(RuntimeError):
|
||||
logs.rotate_mysql(self.root,lambda *_:(_ for _ in ()).throw(RuntimeError('interrupted')),br)
|
||||
self.assertFalse(source.exists());self.assertTrue((self.root/'.runtime/operations/mysql-log-rotation.json').exists())
|
||||
calls=[]
|
||||
def reopen(sql,_):calls.append(sql);source.write_bytes(b'new log')
|
||||
self.assertEqual(logs.rotate_mysql(self.root,reopen,br),1)
|
||||
self.assertEqual(calls,['FLUSH ERROR LOGS;'])
|
||||
archived=list(source.parent.glob('mysql-server.roll-*.log'));self.assertEqual(archived[0].read_bytes(),original)
|
||||
self.assertEqual(source.read_bytes(),b'new log')
|
||||
|
||||
|
||||
class ManagedResourceTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
if not (ROOT / '.runtime/local-config.json').is_file(): raise unittest.SkipTest('Local MySQL test runtime is not configured')
|
||||
def setUp(self):
|
||||
self.temp=tempfile.TemporaryDirectory();self.root=Path(self.temp.name).resolve()
|
||||
cfg=json.loads((ROOT/'.runtime/local-config.json').read_text())
|
||||
br.atomic(self.root/'.runtime/local-config.json',{'mysqlDatabase':cfg['mysqlDatabase']})
|
||||
self.names=[];self.users=[]
|
||||
def tearDown(self):
|
||||
for n in self.names:backup.client('DROP DATABASE IF EXISTS `'+n+'`;')
|
||||
for u in self.users:backup.client("DROP USER IF EXISTS '"+u+"'@'127.0.0.1';")
|
||||
self.temp.cleanup()
|
||||
def create(self,empty=False):
|
||||
name='quality_retention_test_'+uuid.uuid4().hex[:12];self.names.append(name)
|
||||
resources.create(self.root,name,backup.client,empty=empty);return name
|
||||
|
||||
def test_successful_create_and_drop_unregisters_only_its_schema(self):
|
||||
name=self.create();self.assertTrue(resources.load(self.root,name)['ready'])
|
||||
resources.drop(self.root,name,backup.client);self.assertIsNone(resources.load(self.root,name))
|
||||
|
||||
def test_expired_dead_owner_is_cleaned_but_live_owner_is_protected(self):
|
||||
name=self.create();value=resources.load(self.root,name);now=value['expiresAt']+1
|
||||
self.assertEqual(resources.cleanup(self.root,backup.client,now)['testDatabasesRemoved'],0)
|
||||
with patch.object(resources,'active_process',return_value=False):
|
||||
self.assertEqual(resources.cleanup(self.root,backup.client,now)['testDatabasesRemoved'],1)
|
||||
self.assertIsNone(resources.load(self.root,name))
|
||||
|
||||
def test_unregistered_schema_wrong_server_and_wrong_marker_are_never_dropped(self):
|
||||
name=self.create();value=resources.load(self.root,name);now=value['expiresAt']+1
|
||||
value['serverUuid']='different';br.atomic(resources.registry(self.root)/(name+'.json'),value)
|
||||
with patch.object(resources,'active_process',return_value=False):
|
||||
self.assertEqual(resources.cleanup(self.root,backup.client,now)['testDatabasesRemoved'],0)
|
||||
with self.assertRaises(ValueError):resources.drop(self.root,name,backup.client)
|
||||
value['serverUuid']=backup.client('SELECT @@server_uuid;').strip();br.atomic(resources.registry(self.root)/(name+'.json'),value)
|
||||
backup.client('UPDATE `'+name+'`.`'+resources.MARKER+"` SET token='different';")
|
||||
with patch.object(resources,'active_process',return_value=False):
|
||||
self.assertEqual(resources.cleanup(self.root,backup.client,now)['testDatabasesRemoved'],0)
|
||||
with self.assertRaises(ValueError):resources.create(self.root,'quality_platform',backup.client)
|
||||
|
||||
def test_empty_schema_stays_empty_until_migrations_finish_then_seals(self):
|
||||
name=self.create(empty=True)
|
||||
self.assertEqual(backup.client('SHOW TABLES;',name).strip(),'')
|
||||
backup.client('CREATE TABLE business_data(id INT);',name)
|
||||
resources.seal(self.root,name,backup.client)
|
||||
self.assertTrue(resources.marker_matches(resources.load(self.root,name),backup.client))
|
||||
resources.drop(self.root,name,backup.client)
|
||||
|
||||
def test_empty_failed_schema_can_be_dropped_by_its_current_owner(self):
|
||||
name=self.create(empty=True);resources.drop(self.root,name,backup.client)
|
||||
self.assertIsNone(resources.load(self.root,name))
|
||||
|
||||
def test_registered_test_account_is_cleaned_after_expired_schema(self):
|
||||
name=self.create();user='qverify_'+uuid.uuid4().hex[:12];self.users.append(user)
|
||||
backup.client("CREATE USER '"+user+"'@'127.0.0.1' IDENTIFIED BY 'disposable-test-account';")
|
||||
backup.client('GRANT ALL ON `'+name+"`.* TO '"+user+"'@'127.0.0.1';")
|
||||
resources.attach_user(self.root,name,user)
|
||||
with patch.object(resources,'active_process',return_value=False):
|
||||
result=resources.cleanup(self.root,backup.client,resources.load(self.root,name)['expiresAt']+1)
|
||||
self.assertEqual(result['testDatabasesRemoved'],1);self.assertEqual(result['testAccountsRemoved'],1)
|
||||
self.assertIsNone(resources.load(self.root,name))
|
||||
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
@@ -0,0 +1,305 @@
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import tarfile
|
||||
import gzip
|
||||
from contextlib import contextmanager
|
||||
import unittest
|
||||
import uuid
|
||||
from unittest.mock import patch
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0,str(ROOT/'scripts'))
|
||||
import backup_retention as br
|
||||
spec=importlib.util.spec_from_file_location('quality_retention',Path(__file__).with_name('retention.py'))
|
||||
ret=importlib.util.module_from_spec(spec);spec.loader.exec_module(ret)
|
||||
|
||||
|
||||
def good_backup(root, suffix, created='2026-11-09T01:00:00+00:00'):
|
||||
folder=root/'output/backups'/('20261109T010000Z-'+suffix)
|
||||
folder.mkdir(parents=True)
|
||||
m={'formatVersion':3,'status':'ISOLATED_RESTORE_VERIFIED','createdAt':created,
|
||||
'restoreVerifiedAt':created,'queryEvidence':{'releaseId':'current'},
|
||||
'verification':{k:True for k in br.CHECKS}}
|
||||
for name,key in br.ARTIFACTS.items():
|
||||
(folder/name).write_bytes(name.encode());m[key]=br.sha(folder/name)
|
||||
br.atomic(folder/'manifest.json',m)
|
||||
return folder.resolve()
|
||||
|
||||
|
||||
class BackupRotationTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp=tempfile.TemporaryDirectory();self.root=Path(self.temp.name)
|
||||
def tearDown(self):self.temp.cleanup()
|
||||
|
||||
def test_failed_or_corrupted_new_backup_keeps_previous(self):
|
||||
old=good_backup(self.root,'11111111');br.rotate(self.root,old)
|
||||
new=good_backup(self.root,'22222222','2026-11-09T02:00:00+00:00')
|
||||
(new/'database.sql.gz').write_bytes(b'corrupted')
|
||||
with self.assertRaises(ValueError):br.rotate(self.root,new)
|
||||
self.assertTrue(old.is_dir());self.assertEqual(br.latest(self.root)[0],old)
|
||||
|
||||
def test_rotation_keeps_one_and_verifying_older_never_rolls_back_latest(self):
|
||||
old=good_backup(self.root,'11111111');br.rotate(self.root,old)
|
||||
new=good_backup(self.root,'22222222','2026-11-09T02:00:00+00:00');br.rotate(self.root,new)
|
||||
self.assertFalse(old.exists());self.assertEqual(br.latest(self.root)[0],new)
|
||||
older=good_backup(self.root,'33333333');br.rotate(self.root,older)
|
||||
self.assertEqual(br.latest(self.root)[0],new);self.assertFalse(older.exists())
|
||||
|
||||
def test_unknown_and_active_incomplete_folders_are_preserved(self):
|
||||
old=good_backup(self.root,'11111111');unknown=old.parent/'unrelated';unknown.mkdir()
|
||||
incomplete=old.parent/'20261109T010000Z-abcdefab';incomplete.mkdir()
|
||||
br.atomic(incomplete/'.owner.json',{'kind':'quality-backup','root':str(self.root),'id':incomplete.name})
|
||||
new=good_backup(self.root,'22222222','2026-11-09T02:00:00+00:00');br.rotate(self.root,new)
|
||||
self.assertTrue(unknown.exists());self.assertTrue(incomplete.exists())
|
||||
|
||||
def test_shared_lock_excludes_another_process(self):
|
||||
code='import sys;from pathlib import Path;sys.path.insert(0,sys.argv[1]);import backup_retention as b\ntry:\n with b.lock(Path(sys.argv[2]),timeout=0):pass\nexcept b.Busy:sys.exit(17)'
|
||||
with br.lock(self.root):
|
||||
child=subprocess.run([sys.executable,'-c',code,str(ROOT/'scripts'),str(self.root)],capture_output=True)
|
||||
self.assertEqual(child.returncode,17)
|
||||
|
||||
def test_managed_backup_root_cannot_be_redirected_outside_project(self):
|
||||
with tempfile.TemporaryDirectory() as external:
|
||||
folder=good_backup(Path(external),'11111111')
|
||||
(self.root/'output').mkdir();(self.root/'output/backups').symlink_to(folder.parent,target_is_directory=True)
|
||||
with self.assertRaises(ValueError):br.rotate(self.root,folder)
|
||||
self.assertTrue(folder.is_dir())
|
||||
|
||||
def test_interrupted_old_backup_removal_can_resume_without_touching_latest(self):
|
||||
old=good_backup(self.root,'11111111');br.rotate(self.root,old)
|
||||
new=good_backup(self.root,'22222222','2026-11-09T02:00:00+00:00')
|
||||
with patch.object(br.shutil,'rmtree',side_effect=OSError('simulated removal failure')):
|
||||
br.rotate(self.root,new)
|
||||
self.assertEqual(br.latest(self.root)[0],new)
|
||||
self.assertEqual(len(list((self.root/'.runtime/retention/retired').glob('*.json'))),1)
|
||||
br.clean_retired(self.root)
|
||||
self.assertFalse(list((self.root/'.runtime/retention/retired').glob('*.json')))
|
||||
self.assertTrue(new.is_dir())
|
||||
|
||||
|
||||
class DatabaseRetentionTests(unittest.TestCase):
|
||||
"""Real MySQL locking/FK tests in a newly owned, tiny disposable database."""
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
if not (ROOT / '.runtime/local-config.json').is_file(): raise unittest.SkipTest('Local MySQL test runtime is not configured')
|
||||
cfg=json.loads((ROOT/'.runtime/local-config.json').read_text())
|
||||
cls.admin=ret.pymysql.connect(unix_socket=str(ROOT/'.runtime/mysql.sock'),user='root',password=cfg['rootPassword'],autocommit=True)
|
||||
cls.name='quality_retention_test_'+uuid.uuid4().hex[:12]
|
||||
with cls.admin.cursor() as c:
|
||||
c.execute('CREATE DATABASE '+ret.ident(cls.name))
|
||||
c.execute('USE '+ret.ident(cls.name))
|
||||
ddl=[
|
||||
'CREATE TABLE data_release(id varchar(36) PRIMARY KEY,status varchar(24),parent_release_id varchar(36),expected_current_release_id varchar(36),FOREIGN KEY(parent_release_id) REFERENCES data_release(id),FOREIGN KEY(expected_current_release_id) REFERENCES data_release(id))',
|
||||
'CREATE TABLE current_release(dataset_id varchar(20) PRIMARY KEY,release_id varchar(36),FOREIGN KEY(release_id) REFERENCES data_release(id))',
|
||||
'CREATE TABLE transform_build(id varchar(36) PRIMARY KEY,status varchar(24),build_key char(64) UNIQUE)',
|
||||
'CREATE TABLE release_member(release_id varchar(36),transform_build_id varchar(36),FOREIGN KEY(release_id) REFERENCES data_release(id),FOREIGN KEY(transform_build_id) REFERENCES transform_build(id))',
|
||||
'CREATE TABLE fact_black(id int PRIMARY KEY,build_id varchar(36),payload varchar(20),FOREIGN KEY(build_id) REFERENCES transform_build(id))',
|
||||
'CREATE TABLE unknown_build_use(id int PRIMARY KEY,build_id varchar(36),FOREIGN KEY(build_id) REFERENCES transform_build(id))',
|
||||
'CREATE TABLE task(id int PRIMARY KEY,status varchar(24),next_retry_at datetime,retry_count int,max_retries int)',
|
||||
'CREATE TABLE import_run(id int PRIMARY KEY,status varchar(24),expected_current_release_id varchar(36),result_release_id varchar(36),FOREIGN KEY(expected_current_release_id) REFERENCES data_release(id),FOREIGN KEY(result_release_id) REFERENCES data_release(id))',
|
||||
'CREATE TABLE release_action(id int PRIMARY KEY,release_id varchar(36),expected_current_release_id varchar(36),FOREIGN KEY(release_id) REFERENCES data_release(id),FOREIGN KEY(expected_current_release_id) REFERENCES data_release(id))',
|
||||
'CREATE TABLE candidate_reconciliation(id int PRIMARY KEY,release_id varchar(36),baseline_release_id varchar(36),FOREIGN KEY(release_id) REFERENCES data_release(id),FOREIGN KEY(baseline_release_id) REFERENCES data_release(id))',
|
||||
'CREATE TABLE analysis_session(id int PRIMARY KEY,release_id varchar(36))',
|
||||
'CREATE TABLE saved_business_object(id int PRIMARY KEY,release_id varchar(36),FOREIGN KEY(release_id) REFERENCES data_release(id))',
|
||||
'CREATE TABLE source_document(id varchar(36) PRIMARY KEY,source_slot varchar(100))',
|
||||
'CREATE TABLE source_revision(id varchar(36) PRIMARY KEY,source_document_id varchar(36),archive_path varchar(1000),sha256 char(64),parser_config_hash varchar(64),file_size bigint)',
|
||||
'CREATE TABLE source_candidate(id varchar(36) PRIMARY KEY,status varchar(24),source_slot varchar(100),source_path varchar(1000),archive_path varchar(1000),sha256 char(64),parser_config_hash varchar(64),expected_current_release_id varchar(36))',
|
||||
'CREATE TABLE source_upload_draft(id varchar(36) PRIMARY KEY,source_file varchar(100),source_path varchar(1000),sha256 char(64),file_size bigint,candidate_id varchar(36),created_at datetime,expires_at datetime)']
|
||||
for statement in ddl:c.execute(statement+' ENGINE=InnoDB')
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
with cls.admin.cursor() as c:c.execute('DROP DATABASE '+ret.ident(cls.name))
|
||||
cls.admin.close()
|
||||
|
||||
def setUp(self):
|
||||
self.temp=tempfile.TemporaryDirectory();self.root=Path(self.temp.name)
|
||||
self.db=ret.Database(ROOT,self.name);self.db.root=self.root
|
||||
self.db.execute('SET FOREIGN_KEY_CHECKS=0')
|
||||
for table in self.db.tables:self.db.execute('TRUNCATE TABLE '+ret.ident(table))
|
||||
self.db.execute('SET FOREIGN_KEY_CHECKS=1')
|
||||
self.db.execute("INSERT INTO data_release VALUES('old','PUBLISHED',NULL,NULL),('current','PUBLISHED','old','old')")
|
||||
self.db.execute("INSERT INTO current_release VALUES('quality','current')")
|
||||
self.db.execute("INSERT INTO transform_build VALUES('shared','COMPLETED',%s),('obsolete','COMPLETED',%s)",('1'*64,'2'*64))
|
||||
self.db.execute("INSERT INTO release_member VALUES('current','shared'),('old','shared'),('old','obsolete')")
|
||||
self.db.execute("INSERT INTO fact_black VALUES(1,'shared','current data'),(2,'obsolete','old data'),(3,'obsolete','old data')")
|
||||
|
||||
def tearDown(self):self.db.conn.close();self.temp.cleanup()
|
||||
|
||||
def retire_old(self):
|
||||
state={'releases':{'old':1}}
|
||||
with self.db.frozen('current'):self.assertEqual(self.db.release_step('current',state,1800,4000),1)
|
||||
|
||||
def test_running_queued_retry_and_unknown_tasks_prevent_cleanup(self):
|
||||
for status,retry in [('RUNNING',False),('QUEUED',False),('FAILED',True),('FUTURE_STATE',False)]:
|
||||
self.db.execute('DELETE FROM task')
|
||||
self.db.execute('INSERT INTO task VALUES(1,%s,%s,0,3)',(status,datetime_from_future() if retry else None))
|
||||
with self.assertRaises(ret.Busy):
|
||||
with self.db.frozen('current'):self.fail('busy job was ignored')
|
||||
self.assertEqual(len(self.db.rows('SELECT * FROM data_release')),2)
|
||||
|
||||
def test_pending_candidate_and_saved_business_references_are_protected(self):
|
||||
self.db.execute("INSERT INTO data_release VALUES('candidate','CANDIDATE','old','old')")
|
||||
with self.db.frozen('current'):
|
||||
self.assertEqual(self.db.release_step('current',{'releases':{'old':1}},1800,4000),0)
|
||||
self.db.execute("DELETE FROM data_release WHERE id='candidate'")
|
||||
self.db.execute("INSERT INTO saved_business_object VALUES(1,'old')")
|
||||
with self.db.frozen('current'):
|
||||
self.assertEqual(self.db.release_step('current',{'releases':{'old':1}},1800,4000),0)
|
||||
|
||||
def test_analysis_session_reference_is_protected_without_database_fk(self):
|
||||
self.db.execute("INSERT INTO analysis_session VALUES(1,'old')")
|
||||
with self.db.frozen('current'):
|
||||
self.assertEqual(self.db.release_step('current',{'releases':{'old':1}},1800,4000),0)
|
||||
|
||||
def test_pending_source_configuration_preserves_its_old_baseline(self):
|
||||
self.db.execute("INSERT INTO source_candidate VALUES('pending','DISCOVERED','slot','path',NULL,%s,'legacy','old')",('3'*64,))
|
||||
with self.db.frozen('current'):
|
||||
self.assertEqual(self.db.release_step('current',{'releases':{'old':1}},1800,4000),0)
|
||||
|
||||
def test_unknown_business_build_reference_is_preserved(self):
|
||||
self.retire_old();self.db.execute("INSERT INTO unknown_build_use VALUES(1,'obsolete')")
|
||||
with self.db.frozen('current'):
|
||||
self.assertEqual(self.db.build_step({'builds':{'obsolete':1}},1800,4000,1),(0,0))
|
||||
self.assertEqual(self.db.rows("SELECT status FROM transform_build WHERE id='obsolete'")[0]['status'],'COMPLETED')
|
||||
|
||||
def test_changed_publication_and_grace_period_prevent_cleanup(self):
|
||||
with self.assertRaises(ret.Busy):
|
||||
with self.db.frozen('old'):pass
|
||||
with self.db.frozen('current'):
|
||||
self.assertEqual(self.db.release_step('current',{'releases':{'old':3900}},1800,4000),0)
|
||||
|
||||
def test_old_release_removed_but_current_membership_and_rows_preserved(self):
|
||||
self.db.execute("INSERT INTO import_run VALUES(1,'COMPLETED','old','old')")
|
||||
self.db.execute("INSERT INTO release_action VALUES(1,'old',NULL)")
|
||||
self.db.execute("INSERT INTO candidate_reconciliation VALUES(1,'current','old')")
|
||||
self.retire_old()
|
||||
self.assertEqual(self.db.rows("SELECT payload FROM fact_black WHERE build_id='shared'")[0]['payload'],'current data')
|
||||
self.assertEqual(self.db.rows("SELECT * FROM release_member"),[{'release_id':'current','transform_build_id':'shared'}])
|
||||
self.assertIsNone(self.db.rows('SELECT * FROM import_run')[0]['expected_current_release_id'])
|
||||
self.assertIsNone(self.db.rows("SELECT * FROM data_release WHERE id='current'")[0]['parent_release_id'])
|
||||
|
||||
def test_shared_build_is_preserved_and_partial_retirement_cannot_be_reused(self):
|
||||
self.retire_old();state={'builds':{'obsolete':1}}
|
||||
with self.db.frozen('current'):
|
||||
self.assertEqual(self.db.build_step(state,1800,4000,1),(1,0))
|
||||
old=self.db.rows("SELECT * FROM transform_build WHERE id='obsolete'")[0]
|
||||
self.assertEqual(old['status'],'RETIRED');self.assertNotEqual(old['build_key'],'2'*64)
|
||||
self.db.execute("INSERT INTO transform_build VALUES('fresh','COMPLETED',%s)",('2'*64,))
|
||||
for _ in range(4):
|
||||
with self.db.frozen('current'):self.db.build_step(state,1800,4000,1)
|
||||
self.assertFalse(self.db.rows("SELECT * FROM transform_build WHERE id='obsolete'"))
|
||||
self.assertTrue(self.db.rows("SELECT * FROM transform_build WHERE id='shared'"))
|
||||
self.assertTrue(self.db.rows("SELECT * FROM transform_build WHERE id='fresh'"))
|
||||
|
||||
def add_draft(self,candidate=None,where=None,age=25):
|
||||
path=where or self.root/'runtimedata/archive/upload-drafts/upload-fixture.xlsx'
|
||||
path.parent.mkdir(parents=True,exist_ok=True);path.write_bytes(b'workbook bytes')
|
||||
self.db.execute('INSERT INTO source_upload_draft VALUES(%s,%s,%s,%s,%s,%s,TIMESTAMPADD(HOUR,-30,CURRENT_TIMESTAMP),TIMESTAMPADD(HOUR,%s,CURRENT_TIMESTAMP))',('draft','input.xlsx',str(path),br.sha(path),path.stat().st_size,candidate,-age))
|
||||
return path
|
||||
|
||||
def test_expired_unbound_upload_removed_but_fresh_preview_is_kept(self):
|
||||
path=self.add_draft()
|
||||
count,size=ret.cleanup_drafts(self.db,'current',{},time.time(),br)
|
||||
self.assertEqual(count,1);self.assertGreater(size,0);self.assertFalse(path.exists())
|
||||
self.add_draft(age=0)
|
||||
self.assertEqual(ret.cleanup_drafts(self.db,'current',{},time.time(),br),(0,0))
|
||||
|
||||
def test_upload_still_referenced_by_pending_candidate_is_kept(self):
|
||||
path=self.add_draft(candidate='pending')
|
||||
self.db.execute("INSERT INTO source_candidate VALUES('pending','DISCOVERED','slot',%s,NULL,%s,'legacy',NULL)",(str(path),br.sha(path)))
|
||||
self.assertEqual(ret.cleanup_drafts(self.db,'current',{},time.time(),br)[0],1)
|
||||
self.assertTrue(path.exists())
|
||||
|
||||
def test_imported_upload_is_rebound_to_verified_archive_before_cleanup(self):
|
||||
path=self.add_draft(candidate='imported');archive=self.root/'runtimedata/archive/source-revisions/formal.xlsx';archive.parent.mkdir();archive.write_bytes(path.read_bytes())
|
||||
self.db.execute("INSERT INTO source_document VALUES('doc','slot')")
|
||||
self.db.execute("INSERT INTO source_revision VALUES('revision','doc',%s,%s,'legacy',%s)",(str(archive),br.sha(archive),archive.stat().st_size))
|
||||
self.db.execute("INSERT INTO source_candidate VALUES('imported','IMPORTED','slot',%s,NULL,%s,'legacy',NULL)",(str(path),br.sha(path)))
|
||||
self.assertEqual(ret.cleanup_drafts(self.db,'current',{},time.time(),br)[0],1)
|
||||
self.assertFalse(path.exists());self.assertTrue(archive.exists())
|
||||
self.assertEqual(self.db.rows('SELECT source_path FROM source_candidate')[0]['source_path'],str(archive))
|
||||
|
||||
def test_outside_file_is_never_deleted(self):
|
||||
path=self.add_draft(where=self.root/'unmanaged/private.xlsx')
|
||||
ret.cleanup_drafts(self.db,'current',{},time.time(),br)
|
||||
self.assertTrue(path.exists())
|
||||
|
||||
def test_symlink_is_not_followed_or_deleted(self):
|
||||
outside=self.root/'important.xlsx';outside.write_bytes(b'keep this')
|
||||
link=self.root/'runtimedata/archive/upload-drafts/upload-link.xlsx';link.parent.mkdir(parents=True);link.symlink_to(outside)
|
||||
self.db.execute('INSERT INTO source_upload_draft VALUES(%s,%s,%s,%s,%s,NULL,TIMESTAMPADD(HOUR,-30,CURRENT_TIMESTAMP),TIMESTAMPADD(HOUR,-25,CURRENT_TIMESTAMP))',('draft','link.xlsx',str(link),br.sha(outside),outside.stat().st_size))
|
||||
ret.cleanup_drafts(self.db,'current',{},time.time(),br)
|
||||
self.assertTrue(outside.exists());self.assertTrue(link.is_symlink())
|
||||
|
||||
def test_file_changed_after_checksum_is_not_deleted(self):
|
||||
path=self.add_draft();original=ret.fingerprint;calls=0
|
||||
def changed(p):
|
||||
nonlocal calls
|
||||
calls+=1
|
||||
if calls==2:path.write_bytes(b'new file contents')
|
||||
return original(p)
|
||||
with patch.object(ret,'fingerprint',side_effect=changed):
|
||||
count,size=ret.cleanup_drafts(self.db,'current',{},time.time(),br)
|
||||
self.assertEqual((count,size),(0,0));self.assertTrue(path.exists())
|
||||
self.assertTrue(self.db.rows('SELECT * FROM source_upload_draft'))
|
||||
|
||||
def test_new_reference_before_file_unlink_is_rechecked(self):
|
||||
path=self.add_draft();freeze=self.db.frozen;calls=0
|
||||
@contextmanager
|
||||
def insert_reference(expected):
|
||||
nonlocal calls
|
||||
calls+=1
|
||||
with freeze(expected):
|
||||
if calls==2:self.db.execute("INSERT INTO source_candidate VALUES('new','DISCOVERED','slot',%s,NULL,%s,'legacy',NULL)",(str(path),br.sha(path)))
|
||||
yield
|
||||
with patch.object(self.db,'frozen',side_effect=insert_reference):
|
||||
ret.cleanup_drafts(self.db,'current',{},time.time(),br)
|
||||
self.assertTrue(path.exists());self.assertFalse(self.db.rows('SELECT * FROM source_upload_draft'))
|
||||
|
||||
def test_long_running_sql_is_not_interrupted_for_cleanup(self):
|
||||
other=ret.Database(ROOT,self.name)
|
||||
try:
|
||||
other.conn.autocommit(False);other.rows('SELECT * FROM fact_black WHERE id=1 FOR UPDATE')
|
||||
with self.assertRaises(ret.Busy):
|
||||
with self.db.frozen('current'):pass
|
||||
self.assertTrue(other.rows('SELECT 1'));other.conn.rollback()
|
||||
finally:other.conn.close()
|
||||
|
||||
|
||||
def datetime_from_future():
|
||||
from datetime import datetime,timedelta,timezone
|
||||
return datetime.now(timezone.utc).replace(tzinfo=None)+timedelta(hours=1)
|
||||
|
||||
|
||||
class PrivateVerifierTests(unittest.TestCase):
|
||||
def test_small_backup_restores_on_private_server_and_cleans_up(self):
|
||||
import backup
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root=Path(temporary).resolve();(root/'.runtime').mkdir()
|
||||
(root/'.runtime/mysql').symlink_to(ROOT/'.runtime/mysql',target_is_directory=True)
|
||||
base=root/'output/backups';folder=base/'20261109T020000Z-12345678';folder.mkdir(parents=True)
|
||||
source=root/'source';source.mkdir();(source/'config.json').write_text('{}')
|
||||
with tarfile.open(folder/'application.tar.gz','w:gz') as tar:tar.add(source/'config.json',arcname='config.json')
|
||||
with tarfile.open(folder/'archive.tar.gz','w:gz'):pass
|
||||
sql="CREATE TABLE current_release(dataset_id varchar(20),release_id varchar(36)); INSERT INTO current_release VALUES('quality',NULL); CREATE TABLE source_revision(id varchar(36),archive_path varchar(1024),sha256 char(64),file_size bigint);"
|
||||
with gzip.open(folder/'database.sql.gz','wt') as out:out.write(sql)
|
||||
manifest={'formatVersion':3,'createdAt':'2026-11-09T02:00:00+00:00','files':{},'archiveSourceRoot':str(root/'runtimedata/archive'),
|
||||
'applicationFiles':{'config.json':br.sha(source/'config.json')},'requiredApplicationFiles':['config.json'],
|
||||
'tableCounts':{'current_release':1,'source_revision':0},'queryEvidence':{'releaseId':None,'months':[]}}
|
||||
for name,key in br.ARTIFACTS.items():manifest[key]=br.sha(folder/name)
|
||||
br.atomic(folder/'manifest.json',manifest)
|
||||
with patch.object(backup,'ROOT',root),patch.object(backup,'BACKUP_ROOT',base),patch.object(backup,'REQUIRED_APPLICATION_FILES',('config.json',)):
|
||||
result=backup.verify(folder)
|
||||
self.assertEqual(result['status'],'ISOLATED_RESTORE_VERIFIED')
|
||||
self.assertFalse(list((root/'.runtime/retention').glob('verify-*')))
|
||||
self.assertFalse(list((root/'.runtime/retention/scratch').glob('*.json')))
|
||||
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
@@ -0,0 +1,237 @@
|
||||
"""Exercise source retirement against an owned disposable MySQL schema and files."""
|
||||
from contextlib import contextmanager
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import uuid
|
||||
from unittest.mock import patch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / 'scripts'))
|
||||
import backup_retention as br
|
||||
import revision_retention as revisions
|
||||
spec = importlib.util.spec_from_file_location('source_retention_host', Path(__file__).with_name('retention.py'))
|
||||
ret = importlib.util.module_from_spec(spec); spec.loader.exec_module(ret)
|
||||
|
||||
|
||||
class SourceRevisionTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
if not (ROOT / '.runtime/local-config.json').is_file(): raise unittest.SkipTest('Local MySQL test runtime is not configured')
|
||||
cfg = json.loads((ROOT / '.runtime/local-config.json').read_text())
|
||||
cls.admin = ret.pymysql.connect(unix_socket=str(ROOT / '.runtime/mysql.sock'), user='root',
|
||||
password=cfg['rootPassword'], autocommit=True)
|
||||
cls.name = 'quality_retention_test_' + uuid.uuid4().hex[:12]
|
||||
ddl = [
|
||||
'CREATE TABLE data_release(id varchar(36) PRIMARY KEY,status varchar(24))',
|
||||
'CREATE TABLE current_release(dataset_id varchar(20) PRIMARY KEY,release_id varchar(36))',
|
||||
'CREATE TABLE source_document(id varchar(36) PRIMARY KEY,source_slot varchar(100))',
|
||||
"CREATE TABLE source_revision(id varchar(36) PRIMARY KEY,source_document_id varchar(36),archive_path varchar(1000),sha256 char(64),file_size bigint,status varchar(24) DEFAULT 'TRANSFORMED',replaces_revision_id varchar(36),parser_config json)",
|
||||
'CREATE TABLE transform_build(id varchar(36) PRIMARY KEY,source_revision_id varchar(36),FOREIGN KEY(source_revision_id) REFERENCES source_revision(id))',
|
||||
'CREATE TABLE release_member(release_id varchar(36),source_revision_id varchar(36),FOREIGN KEY(source_revision_id) REFERENCES source_revision(id))',
|
||||
'CREATE TABLE fact_black(id int PRIMARY KEY,source_revision_id varchar(36),raw_values json,components json,FOREIGN KEY(source_revision_id) REFERENCES source_revision(id))',
|
||||
'CREATE TABLE source_candidate(id varchar(36) PRIMARY KEY,status varchar(24),sha256 char(64),source_path varchar(1000),archive_path varchar(1000),replacement_revision_id varchar(36))',
|
||||
'CREATE TABLE source_upload_draft(id varchar(36) PRIMARY KEY,source_path varchar(1000))',
|
||||
'CREATE TABLE task(id int PRIMARY KEY,status varchar(24),payload json)',
|
||||
'CREATE TABLE source_change_request(id int PRIMARY KEY,original_revision_id varchar(36),status varchar(24))',
|
||||
'CREATE TABLE saved_context(id int PRIMARY KEY,context_json json,attachment_path varchar(1000))',
|
||||
'CREATE TABLE unknown_reference(id int PRIMARY KEY,revision_ref varchar(36),FOREIGN KEY(revision_ref) REFERENCES source_revision(id))']
|
||||
try:
|
||||
with cls.admin.cursor() as c:
|
||||
c.execute('CREATE DATABASE ' + ret.ident(cls.name)); c.execute('USE ' + ret.ident(cls.name))
|
||||
for statement in ddl:
|
||||
c.execute(statement + ' ENGINE=InnoDB')
|
||||
except BaseException:
|
||||
with cls.admin.cursor() as c: c.execute('DROP DATABASE IF EXISTS ' + ret.ident(cls.name))
|
||||
cls.admin.close(); raise
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
with cls.admin.cursor() as c: c.execute('DROP DATABASE ' + ret.ident(cls.name))
|
||||
cls.admin.close()
|
||||
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory(); self.root = Path(self.temp.name).resolve()
|
||||
self.db = ret.Database(ROOT, self.name); self.db.root = self.root
|
||||
self.db.execute('SET FOREIGN_KEY_CHECKS=0')
|
||||
for table in self.db.tables: self.db.execute('TRUNCATE TABLE ' + ret.ident(table))
|
||||
self.db.execute('SET FOREIGN_KEY_CHECKS=1')
|
||||
self.db.execute("INSERT INTO source_document VALUES('doc','black:2026:01')")
|
||||
self.db.execute("INSERT INTO data_release VALUES('current','PUBLISHED')")
|
||||
self.db.execute("INSERT INTO current_release VALUES('quality','current')")
|
||||
self.current_file = self.add_revision('current-rev', b'current workbook')
|
||||
self.db.execute("INSERT INTO release_member VALUES('current','current-rev')")
|
||||
self.db.execute("INSERT INTO transform_build VALUES('current-build','current-rev')")
|
||||
self.cleaner = revisions.Cleaner(self.db, br, ret.safe_file, ret.fingerprint, ret.Busy)
|
||||
self.state = {}
|
||||
|
||||
def tearDown(self):
|
||||
self.db.conn.close(); self.temp.cleanup()
|
||||
|
||||
def add_revision(self, rid='old-rev-unique', content=b'old workbook'):
|
||||
digest = __import__('hashlib').sha256(content).hexdigest()
|
||||
path = self.root / 'runtimedata/archive/source-revisions/black/2026/01' / (digest + '.xlsx')
|
||||
path.parent.mkdir(parents=True, exist_ok=True); path.write_bytes(content)
|
||||
self.db.execute('INSERT INTO source_revision(id,source_document_id,archive_path,sha256,file_size) VALUES(%s,%s,%s,%s,%s)',
|
||||
(rid, 'doc', str(path), digest, len(content)))
|
||||
return path
|
||||
|
||||
def collect(self, now):
|
||||
return self.cleaner.collect('current', self.state, now)
|
||||
|
||||
def present(self, rid='old-rev-unique'):
|
||||
return bool(self.db.rows('SELECT id FROM source_revision WHERE id=%s', (rid,)))
|
||||
|
||||
def test_unreferenced_revision_waits_grace_then_removes_only_its_owned_file(self):
|
||||
path = self.add_revision()
|
||||
self.assertEqual(self.collect(100)['sourceRevisionsRemoved'], 0)
|
||||
result = self.collect(2000)
|
||||
self.assertEqual(result['sourceRevisionsRemoved'], 1)
|
||||
self.assertEqual(result['sourceArchiveFilesRemoved'], 1)
|
||||
self.assertEqual(result['sourceArchiveBytesRemoved'], len(b'old workbook'))
|
||||
self.assertFalse(self.present()); self.assertFalse(path.exists())
|
||||
self.assertTrue(self.present('current-rev')); self.assertTrue(self.current_file.exists())
|
||||
self.assertTrue(self.db.rows('SELECT * FROM source_document'))
|
||||
|
||||
def test_any_release_build_fact_or_unknown_foreign_key_protects_revision(self):
|
||||
for rid, statement in [
|
||||
('candidate', "INSERT INTO release_member VALUES('candidate','candidate')"),
|
||||
('build', "INSERT INTO transform_build VALUES('old-build','build')"),
|
||||
('fact', "INSERT INTO fact_black VALUES(1,'fact',JSON_OBJECT(),JSON_OBJECT())"),
|
||||
('future', "INSERT INTO unknown_reference VALUES(1,'future')")]:
|
||||
self.add_revision(rid, rid.encode()); self.db.execute(statement)
|
||||
self.collect(100); self.assertEqual(self.collect(2000)['sourceRevisionsRemoved'], 0)
|
||||
for rid in ('candidate', 'build', 'fact', 'future'): self.assertTrue(self.present(rid))
|
||||
|
||||
def test_replacement_withdrawal_and_saved_json_ids_without_fks_are_protected(self):
|
||||
for rid in ('replacement', 'withdrawal', 'lineage', 'saved'): self.add_revision(rid, rid.encode())
|
||||
self.db.execute("INSERT INTO source_candidate VALUES('candidate','DISCOVERED',NULL,NULL,NULL,'replacement')")
|
||||
self.db.execute("INSERT INTO source_change_request VALUES(1,'withdrawal','APPROVED')")
|
||||
self.db.execute("UPDATE source_revision SET replaces_revision_id='lineage' WHERE id='current-rev'")
|
||||
self.db.execute("INSERT INTO saved_context VALUES(1,JSON_OBJECT('sourceRevisionId','saved'),NULL)")
|
||||
self.collect(100); self.assertEqual(self.collect(2000)['sourceRevisionsRemoved'], 0)
|
||||
|
||||
def test_catalog_hash_protects_original_even_with_external_path(self):
|
||||
path = self.add_revision(); digest = br.sha(path)
|
||||
self.db.execute("INSERT INTO source_candidate VALUES('catalog','IMPORTED',%s,'/external/original.xlsx',NULL,NULL)", (digest,))
|
||||
self.collect(100); self.assertEqual(self.collect(2000)['sourceRevisionsRemoved'], 0)
|
||||
self.assertTrue(self.present()); self.assertTrue(path.exists())
|
||||
|
||||
def test_new_reference_resets_continuous_grace(self):
|
||||
self.add_revision(); self.collect(100)
|
||||
self.db.execute("INSERT INTO unknown_reference VALUES(1,'old-rev-unique')"); self.collect(1500)
|
||||
self.db.execute('DELETE FROM unknown_reference')
|
||||
self.assertEqual(self.collect(2000)['sourceRevisionsRemoved'], 0)
|
||||
self.assertEqual(self.collect(3801)['sourceRevisionsRemoved'], 1)
|
||||
|
||||
def test_revision_metadata_change_restarts_grace(self):
|
||||
self.add_revision(); self.collect(100)
|
||||
self.db.execute("UPDATE source_revision SET parser_config=JSON_OBJECT('changed',TRUE) WHERE id='old-rev-unique'")
|
||||
self.assertEqual(self.collect(2000)['sourceRevisionsRemoved'], 0)
|
||||
self.assertEqual(self.collect(3801)['sourceRevisionsRemoved'], 1)
|
||||
|
||||
def test_shared_archive_stays_until_its_other_revision_is_unreferenced(self):
|
||||
path = self.add_revision(content=b'current workbook')
|
||||
self.collect(100); result = self.collect(2000)
|
||||
self.assertEqual(result['sourceRevisionsRemoved'], 1)
|
||||
self.assertEqual(result['sourceArchiveFilesRemoved'], 0)
|
||||
self.assertTrue(path.exists()); self.assertTrue(self.present('current-rev'))
|
||||
|
||||
def test_saved_file_path_protects_file_after_metadata_removal(self):
|
||||
path = self.add_revision()
|
||||
self.db.execute('INSERT INTO saved_context VALUES(1,JSON_OBJECT(\'archivePath\',%s),NULL)', (str(path),))
|
||||
self.collect(100); result = self.collect(2000)
|
||||
self.assertEqual(result['sourceRevisionsRemoved'], 1)
|
||||
self.assertEqual(result['sourceFilesPending'], 1); self.assertTrue(path.exists())
|
||||
self.db.execute('DELETE FROM saved_context')
|
||||
self.assertEqual(self.cleaner.drain('current')[0], 1); self.assertFalse(path.exists())
|
||||
|
||||
def test_external_symlink_mismatched_or_incomplete_files_are_kept(self):
|
||||
for rid in ('outside', 'symlink', 'damaged', 'incomplete'): self.add_revision(rid, rid.encode())
|
||||
external = self.root / 'important.xlsx'; external.write_bytes(b'outside')
|
||||
self.db.execute("UPDATE source_revision SET archive_path=%s WHERE id='outside'", (str(external),))
|
||||
row = self.db.rows("SELECT archive_path FROM source_revision WHERE id='symlink'")[0]
|
||||
link = Path(row['archive_path']); link.unlink(); link.symlink_to(external)
|
||||
row = self.db.rows("SELECT archive_path FROM source_revision WHERE id='damaged'")[0]
|
||||
Path(row['archive_path']).write_bytes(b'changed')
|
||||
self.db.execute("UPDATE source_revision SET status='RECEIVED' WHERE id='incomplete'")
|
||||
self.collect(100); self.assertEqual(self.collect(2000)['sourceRevisionsRemoved'], 0)
|
||||
for rid in ('outside', 'symlink', 'damaged', 'incomplete'): self.assertTrue(self.present(rid))
|
||||
self.assertTrue(external.exists()); self.assertTrue(link.is_symlink())
|
||||
|
||||
def test_reference_added_before_database_delete_is_rechecked(self):
|
||||
path = self.add_revision(); self.collect(100); original = self.db.frozen
|
||||
@contextmanager
|
||||
def add_reference(current):
|
||||
with original(current):
|
||||
self.db.execute("INSERT INTO unknown_reference VALUES(1,'old-rev-unique')")
|
||||
yield
|
||||
with patch.object(self.db, 'frozen', side_effect=add_reference): result = self.collect(2000)
|
||||
self.assertEqual(result['sourceRevisionsRemoved'], 0)
|
||||
self.assertTrue(path.exists()); self.assertTrue(self.present())
|
||||
|
||||
def test_file_changed_after_hash_is_kept(self):
|
||||
path = self.add_revision(); self.collect(100); original = self.db.frozen
|
||||
@contextmanager
|
||||
def change_file(current):
|
||||
path.write_bytes(b'file edited during preflight')
|
||||
with original(current): yield
|
||||
with patch.object(self.db, 'frozen', side_effect=change_file): result = self.collect(2000)
|
||||
self.assertEqual(result['sourceRevisionsRemoved'], 0)
|
||||
self.assertTrue(path.exists()); self.assertTrue(self.present())
|
||||
|
||||
def test_crash_after_database_commit_resumes_registered_file_removal(self):
|
||||
path = self.add_revision(); self.collect(100)
|
||||
with patch.object(self.cleaner, 'drain', side_effect=RuntimeError('interrupted')):
|
||||
with self.assertRaises(RuntimeError): self.collect(2000)
|
||||
self.assertFalse(self.present()); self.assertTrue(path.exists())
|
||||
self.assertEqual(self.cleaner.drain('current')[0], 1)
|
||||
self.assertFalse(path.exists()); self.assertTrue(self.current_file.exists())
|
||||
|
||||
def test_database_rollback_keeps_file_and_discards_only_journal(self):
|
||||
path = self.add_revision(); self.collect(100); original = self.db.execute
|
||||
def rollback(sql, args=()):
|
||||
result = original(sql, args)
|
||||
if sql.startswith('DELETE FROM source_revision'): raise RuntimeError('rollback')
|
||||
return result
|
||||
with patch.object(self.db, 'execute', side_effect=rollback):
|
||||
with self.assertRaises(RuntimeError): self.collect(2000)
|
||||
self.assertTrue(self.present()); self.assertTrue(path.exists())
|
||||
self.assertEqual(self.cleaner.drain('current'), (0, 0))
|
||||
self.assertFalse(list(self.cleaner.journals.glob('*.json')))
|
||||
|
||||
def test_new_path_reference_before_unlink_protects_file(self):
|
||||
path = self.add_revision(); self.collect(100); original = self.cleaner.drain
|
||||
def add_reference(current, limit=5):
|
||||
self.db.execute('INSERT INTO source_upload_draft VALUES(\'new\',%s)', (str(path),))
|
||||
return original(current, limit)
|
||||
with patch.object(self.cleaner, 'drain', side_effect=add_reference): result = self.collect(2000)
|
||||
self.assertEqual(result['sourceRevisionsRemoved'], 1)
|
||||
self.assertEqual(result['sourceArchiveFilesRemoved'], 0); self.assertTrue(path.exists())
|
||||
|
||||
def test_running_task_or_changed_publication_defers_without_deleting(self):
|
||||
path = self.add_revision(); self.collect(100)
|
||||
self.db.execute("INSERT INTO task VALUES(1,'RUNNING',NULL)")
|
||||
with self.assertRaises(ret.Busy): self.collect(2000)
|
||||
self.assertTrue(self.present()); self.assertTrue(path.exists())
|
||||
self.db.execute('DELETE FROM task')
|
||||
self.db.execute("UPDATE current_release SET release_id='different'")
|
||||
with self.assertRaises(ret.Busy): self.collect(2000)
|
||||
self.assertTrue(self.present()); self.assertTrue(path.exists())
|
||||
|
||||
def test_unsafe_registry_and_unknown_stray_files_are_not_deleted(self):
|
||||
path = self.add_revision(); self.collect(100)
|
||||
self.cleaner.journals.mkdir(parents=True)
|
||||
br.atomic(self.cleaner.journals / ('1' * 32 + '.json'), {'kind': 'quality-source-retirement'})
|
||||
unknown = path.with_name('unregistered.xlsx'); unknown.write_bytes(b'keep')
|
||||
self.cleaner.drain('current'); self.assertTrue(unknown.exists()); self.assertTrue(path.exists())
|
||||
with self.cleaner.quick_checks():
|
||||
self.cleaner.reference_deadline = time.monotonic() - 1
|
||||
with self.assertRaises(ret.Busy): self.cleaner.exists('source_revision', 'id=%s', ('old-rev-unique',))
|
||||
|
||||
|
||||
if __name__ == '__main__': unittest.main()
|
||||
@@ -0,0 +1,116 @@
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
|
||||
spec = importlib.util.spec_from_file_location('quality_macos_service', Path(__file__).with_name('service.py'))
|
||||
service = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(service)
|
||||
|
||||
|
||||
class Fixture(service.Guardian):
|
||||
def command(self, role):
|
||||
return ['/bin/sleep', '90' if role == 'mysql' else '91']
|
||||
|
||||
def occupied(self, role):
|
||||
return False
|
||||
|
||||
def environment(self):
|
||||
return os.environ.copy()
|
||||
|
||||
def db_ready(self):
|
||||
return self.ready
|
||||
|
||||
def backend_ready(self):
|
||||
return service.matches(self.records.get('backend')) is True
|
||||
|
||||
|
||||
class GuardianTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.temp.name)
|
||||
(self.root / '.runtime').mkdir()
|
||||
self.g = Fixture({'root': str(self.root)}, self.root / 'guardian')
|
||||
self.g.value = {'mysqlPort': 59998}
|
||||
self.g.ready = False
|
||||
self.external = []
|
||||
|
||||
def tearDown(self):
|
||||
for p in list(self.g.children.values()) + self.external:
|
||||
if p.poll() is None:
|
||||
p.terminate()
|
||||
p.wait(timeout=10)
|
||||
self.temp.cleanup()
|
||||
|
||||
def test_database_must_be_ready_before_backend_starts(self):
|
||||
self.g.tick()
|
||||
self.assertIn('mysql', self.g.records)
|
||||
self.assertNotIn('backend', self.g.records)
|
||||
self.g.ready = True
|
||||
self.g.tick()
|
||||
self.assertTrue(service.matches(self.g.records['backend']))
|
||||
|
||||
def test_abnormal_child_exit_is_restarted_without_duplicate_mysql(self):
|
||||
self.g.ready = True
|
||||
self.g.tick()
|
||||
mysql = self.g.records['mysql']['pid']
|
||||
old = self.g.children['backend']
|
||||
old.kill()
|
||||
old.wait()
|
||||
self.g.tick()
|
||||
self.assertNotIn('backend', self.g.records)
|
||||
self.assertEqual(self.g.records['mysql']['pid'], mysql)
|
||||
self.g.next_start['backend'] = 0
|
||||
self.g.tick()
|
||||
self.assertNotEqual(self.g.records['backend']['pid'], old.pid)
|
||||
self.assertEqual(self.g.restarts, {'mysql': 1, 'backend': 2})
|
||||
|
||||
def test_database_exit_is_restarted_before_missing_backend(self):
|
||||
self.g.tick()
|
||||
old = self.g.children['mysql']
|
||||
old.kill()
|
||||
old.wait()
|
||||
self.g.tick()
|
||||
self.assertNotIn('mysql', self.g.records)
|
||||
self.g.next_start['mysql'] = 0
|
||||
self.g.tick()
|
||||
self.assertNotEqual(self.g.records['mysql']['pid'], old.pid)
|
||||
self.assertNotIn('backend', self.g.records)
|
||||
|
||||
def test_existing_process_is_adopted_and_unknown_process_is_not_replaced(self):
|
||||
command = self.g.command('mysql')
|
||||
p = subprocess.Popen(command, start_new_session=True)
|
||||
self.external.append(p)
|
||||
info = service.record(p.pid, command)
|
||||
service.atomic(self.root / '.runtime/mysql.process.json', {'pid': p.pid, 'started': info['started']})
|
||||
self.assertTrue(self.g.ensure('mysql'))
|
||||
self.assertEqual(self.g.restarts['mysql'], 0)
|
||||
self.assertIsNone(service.matches(info, self.g.command('backend')))
|
||||
self.assertIs(p.poll(), None)
|
||||
|
||||
def test_missing_data_marker_never_initializes_database(self):
|
||||
service.atomic(self.root / '.runtime/local-config.json', {'databaseReady': True})
|
||||
real = service.Guardian({'root': str(self.root), 'jarSha256': '0'*64}, self.root / 'guardian')
|
||||
with self.assertRaises(RuntimeError):
|
||||
real.preflight()
|
||||
self.assertFalse((self.root / '.runtime/mysql-data').exists())
|
||||
|
||||
def test_pinned_jar_mismatch_is_rejected(self):
|
||||
r = self.root / '.runtime'
|
||||
(r / 'mysql-data').mkdir()
|
||||
(r / 'mysql-data/auto.cnf').write_text('existing marker')
|
||||
(r / 'backend-builds').mkdir()
|
||||
(r / 'backend-builds' / ('0'*64+'.jar')).write_bytes(b'unaccepted')
|
||||
service.atomic(r / 'local-config.json', {'databaseReady': True})
|
||||
real = service.Guardian({'root': str(self.root), 'jarSha256': '0'*64}, self.root / 'guardian')
|
||||
with self.assertRaises(RuntimeError):
|
||||
real.preflight()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user