refactor: implement photo processing pipeline and modularize photo handling

- Introduced a comprehensive photo processing pipeline that integrates various processing steps including thumbnail generation, EXIF data extraction, and tone analysis.
- Modularized the photo handling logic into distinct files for better organization and maintainability.
- Added support for Live Photo detection and processing.
- Implemented a global logger system to streamline logging across different processing stages.
- Enhanced caching mechanisms for reusing existing data, improving performance and efficiency.

Signed-off-by: Innei <tukon479@gmail.com>
This commit is contained in:
Innei
2025-06-22 23:45:09 +08:00
parent 123bd9bf9e
commit efe2875636
10 changed files with 911 additions and 266 deletions

View File

@@ -129,6 +129,8 @@
"exif.white.balance.shift.gm": "白平衡偏移 (绿色 - 品红色)",
"exif.white.balance.title": "白平衡",
"gallery.built.at": "构建于 ",
"gallery.photos_one": "{{count}} 张照片",
"gallery.photos_other": "{{count}} 张照片",
"histogram.blue": "蓝",
"histogram.channel": "通道",
"histogram.green": "绿",

View File

@@ -136,9 +136,9 @@ export class PhotoGalleryBuilder {
`存储中找到 ${imageObjects.length} 张照片,实际需要处理 ${tasksToProcess.length}`,
)
// 如果没有任务需要处理直接使用现有的manifest
// 如果没有任务需要处理,直接使用现有的 manifest
if (tasksToProcess.length === 0) {
logger.main.info('💡 没有需要处理的照片使用现有manifest')
logger.main.info('💡 没有需要处理的照片,使用现有 manifest')
manifest.push(
...existingManifestItems.filter((item) => s3ImageKeys.has(item.s3Key)),
)
@@ -150,7 +150,7 @@ export class PhotoGalleryBuilder {
// 根据配置和实际任务数量选择处理模式
const { useClusterMode } = this.config.performance.worker
// 如果实际任务数量较少则不使用cluster模式
// 如果实际任务数量较少,则不使用 cluster 模式
const shouldUseCluster =
useClusterMode && tasksToProcess.length >= concurrency * 2
@@ -251,7 +251,7 @@ export class PhotoGalleryBuilder {
}
}
// 添加未处理但仍然存在的照片到manifest
// 添加未处理但仍然存在的照片到 manifest
for (const [key, item] of existingManifestMap) {
if (s3ImageKeys.has(key) && !manifest.some((m) => m.s3Key === key)) {
manifest.push(item)
@@ -371,7 +371,7 @@ export class PhotoGalleryBuilder {
/**
* 筛选出实际需要处理的图片
* @param imageObjects 存储中的图片对象列表
* @param existingManifestMap 现有manifest的映射
* @param existingManifestMap 现有 manifest 的映射
* @param options 构建选项
* @returns 实际需要处理的图片数组
*/

View File

@@ -0,0 +1,104 @@
# 照片处理模块
这个模块包含了照片处理的核心逻辑,采用了模块化设计,将不同的处理逻辑分离到不同的文件中。
## 模块结构
### 核心文件
- **`processor.ts`** - 主要的照片处理入口点
- **`image-pipeline.ts`** - 图片处理管道,整合所有处理步骤
- **`cache-manager.ts`** - 缓存管理处理缩略图、EXIF、影调分析等数据的复用
- **`live-photo-handler.ts`** - Live Photo 检测和处理
- **`logger-adapter.ts`** - Logger 适配器,实现适配器模式
- **`info-extractor.ts`** - 照片信息提取
### 设计模式
#### 适配器模式 (Adapter Pattern)
使用适配器模式来统一不同的 Logger 接口:
```typescript
// 通用 Logger 接口
interface PhotoLogger {
info(message: string, ...args: any[]): void
warn(message: string, ...args: any[]): void
error(message: string, error?: any): void
success(message: string, ...args: any[]): void
}
// 适配器实现
class CompatibleLoggerAdapter implements PhotoLogger {
// 既实现通用接口,又兼容原有的 ConsolaInstance
}
```
#### 管道模式 (Pipeline Pattern)
图片处理采用管道模式,按步骤处理:
1. 预处理图片数据
2. 创建 Sharp 实例
3. 处理缩略图和 blurhash
4. 处理 EXIF 数据
5. 处理影调分析
6. 提取照片信息
7. 处理 Live Photo
8. 构建照片清单项
### 主要改进
1. **模块化分离**: 将不同的处理逻辑分离到专门的模块中
2. **Logger 适配器**: 不再通过参数传递 logger使用全局 logger 适配器
3. **缓存管理**: 统一管理各种数据的缓存和复用逻辑
4. **Live Photo 处理**: 专门的模块处理 Live Photo 检测和匹配
5. **类型安全**: 完善的 TypeScript 类型定义
### 使用方法
#### 基本使用
```typescript
import { processPhoto, setGlobalLoggers, createPhotoProcessingLoggers } from './index.js'
// 设置全局 logger
const loggers = createPhotoProcessingLoggers(workerId, baseLogger)
setGlobalLoggers(loggers)
// 处理照片
const result = await processPhoto(obj, index, workerId, totalImages, existingManifestMap, livePhotoMap, options)
```
#### 单独使用各个模块
```typescript
import {
processLivePhoto,
processThumbnailAndBlurhash,
processExifData
} from './index.js'
// Live Photo 处理
const livePhotoResult = processLivePhoto(photoKey, livePhotoMap)
// 缩略图处理
const thumbnailResult = await processThumbnailAndBlurhash(imageBuffer, photoId, width, height, existingItem, options)
// EXIF 处理
const exifData = await processExifData(imageBuffer, rawImageBuffer, photoKey, existingItem, options)
```
### 扩展性
新的模块化设计使得扩展新功能变得更加容易:
1. 添加新的处理步骤只需要在管道中插入新的函数
2. 添加新的缓存类型只需要扩展 `cache-manager.ts`
3. 添加新的 Logger 适配器只需要实现 `PhotoLogger` 接口
### 性能优化
1. **缓存复用**: 智能复用现有的缩略图、EXIF、影调分析数据
2. **Sharp 实例复用**: 在处理管道中复用 Sharp 实例
3. **条件处理**: 只在需要时处理特定的数据类型

View File

@@ -0,0 +1,202 @@
import path from 'node:path'
import { workdir } from '@afilmory/builder/path.js'
import type sharp from 'sharp'
import { HEIC_FORMATS } from '../constants/index.js'
import { extractExifData } from '../image/exif.js'
import { calculateHistogramAndAnalyzeTone } from '../image/histogram.js'
import {
generateThumbnailAndBlurhash,
thumbnailExists,
} from '../image/thumbnail.js'
import type {
PhotoManifestItem,
PickedExif,
ToneAnalysis,
} from '../types/photo.js'
import { getGlobalLoggers } from './logger-adapter.js'
import type { PhotoProcessorOptions } from './processor.js'
export interface ThumbnailResult {
thumbnailUrl: string
thumbnailBuffer: Buffer
blurhash: string
}
export interface CacheableData {
thumbnail?: ThumbnailResult
exif?: PickedExif
toneAnalysis?: ToneAnalysis
}
/**
* 处理缩略图和 blurhash
* 优先复用现有数据,如果不存在或需要强制更新则重新生成
*/
export async function processThumbnailAndBlurhash(
imageBuffer: Buffer,
photoId: string,
width: number,
height: number,
existingItem: PhotoManifestItem | undefined,
options: PhotoProcessorOptions,
): Promise<ThumbnailResult> {
const loggers = getGlobalLoggers()
// 检查是否可以复用现有数据
if (
!options.isForceMode &&
!options.isForceThumbnails &&
existingItem?.blurhash &&
(await thumbnailExists(photoId))
) {
try {
const fs = await import('node:fs/promises')
const thumbnailPath = path.join(
workdir,
'public/thumbnails',
`${photoId}.webp`,
)
const thumbnailBuffer = await fs.readFile(thumbnailPath)
const thumbnailUrl = `/thumbnails/${photoId}.webp`
loggers.blurhash.info(`复用现有 blurhash: ${photoId}`)
loggers.thumbnail.info(`复用现有缩略图:${photoId}`)
return {
thumbnailUrl,
thumbnailBuffer,
blurhash: existingItem.blurhash,
}
} catch (error) {
loggers.thumbnail.warn(`读取现有缩略图失败,重新生成:${photoId}`, error)
// 继续执行生成逻辑
}
}
// 生成新的缩略图和 blurhash
const result = await generateThumbnailAndBlurhash(
imageBuffer,
photoId,
width,
height,
options.isForceMode || options.isForceThumbnails,
{
thumbnail: loggers.thumbnail.originalLogger,
blurhash: loggers.blurhash.originalLogger,
},
)
return {
thumbnailUrl: result.thumbnailUrl!,
thumbnailBuffer: result.thumbnailBuffer!,
blurhash: result.blurhash!,
}
}
/**
* 处理 EXIF 数据
* 优先复用现有数据,如果不存在或需要强制更新则重新提取
*/
export async function processExifData(
imageBuffer: Buffer,
rawImageBuffer: Buffer | undefined,
photoKey: string,
existingItem: PhotoManifestItem | undefined,
options: PhotoProcessorOptions,
): Promise<PickedExif | null> {
const loggers = getGlobalLoggers()
// 检查是否可以复用现有数据
if (!options.isForceMode && !options.isForceManifest && existingItem?.exif) {
const photoId = path.basename(photoKey, path.extname(photoKey))
loggers.exif.info(`复用现有 EXIF 数据:${photoId}`)
return existingItem.exif
}
// 提取新的 EXIF 数据
const ext = path.extname(photoKey).toLowerCase()
const originalBuffer = HEIC_FORMATS.has(ext) ? rawImageBuffer : undefined
return await extractExifData(
imageBuffer,
originalBuffer,
loggers.exif.originalLogger,
)
}
/**
* 处理影调分析
* 优先复用现有数据,如果不存在或需要强制更新则重新计算
*/
export async function processToneAnalysis(
sharpInstance: sharp.Sharp,
photoKey: string,
existingItem: PhotoManifestItem | undefined,
options: PhotoProcessorOptions,
): Promise<ToneAnalysis | null> {
const loggers = getGlobalLoggers()
// 检查是否可以复用现有数据
if (
!options.isForceMode &&
!options.isForceManifest &&
existingItem?.toneAnalysis
) {
const photoId = path.basename(photoKey, path.extname(photoKey))
loggers.tone.info(`复用现有影调分析:${photoId}`)
return existingItem.toneAnalysis
}
// 计算新的影调分析
return await calculateHistogramAndAnalyzeTone(
sharpInstance,
loggers.tone.originalLogger,
)
}
/**
* 检查是否需要处理照片
* 考虑文件更新状态和缓存存在性
*/
export async function shouldProcessPhoto(
photoKey: string,
existingItem: PhotoManifestItem | undefined,
obj: { LastModified?: Date; ETag?: string },
options: PhotoProcessorOptions,
): Promise<{ shouldProcess: boolean; reason: string }> {
const photoId = path.basename(photoKey, path.extname(photoKey))
// 强制模式下总是处理
if (options.isForceMode) {
return { shouldProcess: true, reason: '强制模式' }
}
// 新照片总是需要处理
if (!existingItem) {
return { shouldProcess: true, reason: '新照片' }
}
// 检查文件是否更新
const fileNeedsUpdate =
existingItem.lastModified !== obj.LastModified?.toISOString()
if (fileNeedsUpdate || options.isForceManifest) {
return {
shouldProcess: true,
reason: fileNeedsUpdate ? '文件已更新' : '强制更新清单',
}
}
// 检查缩略图是否存在
const hasThumbnail = await thumbnailExists(photoId)
if (!hasThumbnail || options.isForceThumbnails) {
return {
shouldProcess: true,
reason: options.isForceThumbnails ? '强制重新生成缩略图' : '缩略图缺失',
}
}
return { shouldProcess: false, reason: '无需处理' }
}

View File

@@ -0,0 +1,271 @@
import type { _Object } from '@aws-sdk/client-s3'
import sharp from 'sharp'
import {
convertBmpToJpegSharpInstance,
getImageMetadataWithSharp,
isBitmap,
preprocessImageBuffer,
} from '../image/processor.js'
import { defaultStorageManager } from '../storage/manager.js'
import type { PhotoManifestItem } from '../types/photo.js'
import {
processExifData,
processThumbnailAndBlurhash,
processToneAnalysis,
shouldProcessPhoto,
} from './cache-manager.js'
import { extractPhotoInfo } from './info-extractor.js'
import { processLivePhoto } from './live-photo-handler.js'
import { getGlobalLoggers } from './logger-adapter.js'
import type { PhotoProcessorOptions } from './processor.js'
export interface ProcessedImageData {
sharpInstance: sharp.Sharp
imageBuffer: Buffer
metadata: { width: number; height: number }
}
export interface PhotoProcessingContext {
photoKey: string
photoId: string
obj: _Object
existingItem: PhotoManifestItem | undefined
livePhotoMap: Map<string, _Object>
options: PhotoProcessorOptions
}
/**
* 预处理图片数据
* 包括获取原始数据、格式转换、BMP 处理等
*/
export async function preprocessImage(
photoKey: string,
): Promise<{ rawBuffer: Buffer; processedBuffer: Buffer } | null> {
const loggers = getGlobalLoggers()
try {
// 获取图片数据
const rawImageBuffer = await defaultStorageManager.getFile(photoKey)
if (!rawImageBuffer) {
loggers.image.error(`无法获取图片数据:${photoKey}`)
return null
}
// 预处理图片(处理 HEIC/HEIF 格式)
let imageBuffer: Buffer
try {
imageBuffer = await preprocessImageBuffer(
rawImageBuffer,
photoKey,
loggers.image.originalLogger,
)
} catch (error) {
loggers.image.error(`预处理图片失败:${photoKey}`, error)
return null
}
return {
rawBuffer: rawImageBuffer,
processedBuffer: imageBuffer,
}
} catch (error) {
loggers.image.error(`图片预处理失败:${photoKey}`, error)
return null
}
}
/**
* 处理图片并创建 Sharp 实例
* 包括 BMP 转换和元数据提取
*/
export async function processImageWithSharp(
imageBuffer: Buffer,
photoKey: string,
): Promise<ProcessedImageData | null> {
const loggers = getGlobalLoggers()
try {
// 创建 Sharp 实例,复用于多个操作
let sharpInstance = sharp(imageBuffer)
let processedBuffer = imageBuffer
// 处理 BMP
if (isBitmap(imageBuffer)) {
try {
// Convert the BMP image to JPEG format and create a new Sharp instance for the converted image.
sharpInstance = await convertBmpToJpegSharpInstance(
imageBuffer,
loggers.image.originalLogger,
)
// Update the image buffer to reflect the new JPEG data from the Sharp instance.
processedBuffer = await sharpInstance.toBuffer()
} catch (error) {
loggers.image.error(`转换 BMP 失败:${photoKey}`, error)
return null
}
}
// 获取图片元数据(复用 Sharp 实例)
const metadata = await getImageMetadataWithSharp(
sharpInstance,
loggers.image.originalLogger,
)
if (!metadata) {
loggers.image.error(`获取图片元数据失败:${photoKey}`)
return null
}
return {
sharpInstance,
imageBuffer: processedBuffer,
metadata,
}
} catch (error) {
loggers.image.error(`Sharp 处理失败:${photoKey}`, error)
return null
}
}
/**
* 完整的照片处理管道
* 整合所有处理步骤
*/
export async function executePhotoProcessingPipeline(
context: PhotoProcessingContext,
): Promise<PhotoManifestItem | null> {
const { photoKey, photoId, obj, existingItem, livePhotoMap, options } =
context
const loggers = getGlobalLoggers()
try {
// 1. 预处理图片
const imageData = await preprocessImage(photoKey)
if (!imageData) return null
// 2. 处理图片并创建 Sharp 实例
const processedData = await processImageWithSharp(
imageData.processedBuffer,
photoKey,
)
if (!processedData) return null
const { sharpInstance, imageBuffer, metadata } = processedData
// 3. 处理缩略图和 blurhash
const thumbnailResult = await processThumbnailAndBlurhash(
imageBuffer,
photoId,
metadata.width,
metadata.height,
existingItem,
options,
)
// 4. 处理 EXIF 数据
const exifData = await processExifData(
imageBuffer,
imageData.rawBuffer,
photoKey,
existingItem,
options,
)
// 5. 处理影调分析
const toneAnalysis = await processToneAnalysis(
sharpInstance,
photoKey,
existingItem,
options,
)
// 6. 提取照片信息
const photoInfo = extractPhotoInfo(
photoKey,
exifData,
loggers.image.originalLogger,
)
// 7. 处理 Live Photo
const livePhotoResult = processLivePhoto(photoKey, livePhotoMap)
// 8. 构建照片清单项
const aspectRatio = metadata.width / metadata.height
const photoItem: PhotoManifestItem = {
id: photoId,
title: photoInfo.title,
description: photoInfo.description,
dateTaken: photoInfo.dateTaken,
views: photoInfo.views,
tags: photoInfo.tags,
originalUrl: defaultStorageManager.generatePublicUrl(photoKey),
thumbnailUrl: thumbnailResult.thumbnailUrl,
blurhash: thumbnailResult.blurhash,
width: metadata.width,
height: metadata.height,
aspectRatio,
s3Key: photoKey,
lastModified: obj.LastModified?.toISOString() || new Date().toISOString(),
size: obj.Size || 0,
exif: exifData,
toneAnalysis,
// Live Photo 相关字段
isLivePhoto: livePhotoResult.isLivePhoto,
livePhotoVideoUrl: livePhotoResult.livePhotoVideoUrl,
livePhotoVideoS3Key: livePhotoResult.livePhotoVideoS3Key,
}
loggers.image.success(`✅ 处理完成:${photoKey}`)
return photoItem
} catch (error) {
loggers.image.error(`❌ 处理管道失败:${photoKey}`, error)
return null
}
}
/**
* 决定是否需要处理照片并返回处理结果
*/
export async function processPhotoWithPipeline(
context: PhotoProcessingContext,
): Promise<{
item: PhotoManifestItem | null
type: 'new' | 'processed' | 'skipped' | 'failed'
}> {
const { photoKey, existingItem, obj, options } = context
const loggers = getGlobalLoggers()
// 检查是否需要处理
const { shouldProcess, reason } = await shouldProcessPhoto(
photoKey,
existingItem,
obj,
options,
)
if (!shouldProcess) {
loggers.image.info(`⏭️ 跳过处理 (${reason}): ${photoKey}`)
return { item: existingItem!, type: 'skipped' }
}
// 记录处理原因
const isNewPhoto = !existingItem
if (isNewPhoto) {
loggers.image.info(`🆕 新照片:${photoKey}`)
} else {
loggers.image.info(`🔄 更新照片 (${reason})${photoKey}`)
}
// 执行处理管道
const processedItem = await executePhotoProcessingPipeline(context)
if (!processedItem) {
return { item: null, type: 'failed' }
}
return {
item: processedItem,
type: isNewPhoto ? 'new' : 'processed',
}
}

View File

@@ -0,0 +1,42 @@
// 主要处理器
export type { PhotoProcessorOptions } from './processor.js'
export { processPhoto } from './processor.js'
// 图片处理管道
export type {
PhotoProcessingContext,
ProcessedImageData,
} from './image-pipeline.js'
export {
executePhotoProcessingPipeline,
preprocessImage,
processImageWithSharp,
processPhotoWithPipeline,
} from './image-pipeline.js'
// 缓存管理
export type { CacheableData, ThumbnailResult } from './cache-manager.js'
export {
processExifData,
processThumbnailAndBlurhash,
processToneAnalysis,
shouldProcessPhoto,
} from './cache-manager.js'
// Live Photo 处理
export type { LivePhotoResult } from './live-photo-handler.js'
export { createLivePhotoMap, processLivePhoto } from './live-photo-handler.js'
// 信息提取
export { extractPhotoInfo } from './info-extractor.js'
// Logger 适配器
export type { PhotoLogger, PhotoProcessingLoggers } from './logger-adapter.js'
export {
CompatibleLoggerAdapter,
createPhotoProcessingLoggers,
getGlobalLoggers,
LoggerAdapter,
setGlobalLoggers,
WorkerLoggerAdapter,
} from './logger-adapter.js'

View File

@@ -13,11 +13,11 @@ export function extractPhotoInfo(
): PhotoInfo {
const log = imageLogger
log?.debug(`提取照片信息: ${key}`)
log?.debug(`提取照片信息${key}`)
const fileName = path.basename(key, path.extname(key))
// 尝试从文件名解析信息,格式示例: "2024-01-15_城市夜景_1250views"
// 尝试从文件名解析信息,格式示例"2024-01-15_城市夜景_1250views"
let title = fileName
let dateTaken = new Date().toISOString()
let views = 0

View File

@@ -0,0 +1,115 @@
import type { _Object } from '@aws-sdk/client-s3'
import type { StorageObject } from '../storage/interfaces.js'
import { defaultStorageManager } from '../storage/manager.js'
import { getGlobalLoggers } from './logger-adapter.js'
export interface LivePhotoResult {
isLivePhoto: boolean
livePhotoVideoUrl?: string
livePhotoVideoS3Key?: string
}
/**
* 检测并处理 Live Photo
* @param photoKey 照片的 S3 key
* @param livePhotoMap Live Photo 映射表
* @returns Live Photo 处理结果
*/
export function processLivePhoto(
photoKey: string,
livePhotoMap: Map<string, _Object | StorageObject>,
): LivePhotoResult {
const loggers = getGlobalLoggers()
const livePhotoVideo = livePhotoMap.get(photoKey)
const isLivePhoto = !!livePhotoVideo
if (!isLivePhoto) {
return { isLivePhoto: false }
}
// 处理不同类型的视频对象
let videoKey: string
if ('Key' in livePhotoVideo && livePhotoVideo.Key) {
// _Object 类型
videoKey = livePhotoVideo.Key
} else if ('key' in livePhotoVideo && livePhotoVideo.key) {
// StorageObject 类型
videoKey = livePhotoVideo.key
} else {
return { isLivePhoto: false }
}
const livePhotoVideoUrl = defaultStorageManager.generatePublicUrl(videoKey)
loggers.image.info(`📱 检测到 Live Photo${photoKey} -> ${videoKey}`)
return {
isLivePhoto: true,
livePhotoVideoUrl,
livePhotoVideoS3Key: videoKey,
}
}
/**
* 创建 Live Photo 映射表 (兼容 _Object 类型)
* 根据文件名匹配 Live Photo 的照片和视频文件
* @param objects S3 对象列表
* @returns Live Photo 映射表
*/
export function createLivePhotoMap(objects: _Object[]): Map<string, _Object>
/**
* 创建 Live Photo 映射表 (兼容 StorageObject 类型)
* 根据文件名匹配 Live Photo 的照片和视频文件
* @param objects 存储对象列表
* @returns Live Photo 映射表
*/
export function createLivePhotoMap(
objects: StorageObject[],
): Map<string, StorageObject>
export function createLivePhotoMap(
objects: _Object[] | StorageObject[],
): Map<string, _Object | StorageObject> {
const livePhotoMap = new Map<string, _Object | StorageObject>()
// 分离照片和视频文件
const photos: (_Object | StorageObject)[] = []
const videos: (_Object | StorageObject)[] = []
for (const obj of objects) {
// 获取 key兼容两种类型
const key = 'Key' in obj ? obj.Key : (obj as StorageObject).key
if (!key) continue
const ext = key.toLowerCase().split('.').pop()
if (ext && ['jpg', 'jpeg', 'heic', 'heif', 'png', 'webp'].includes(ext)) {
photos.push(obj)
} else if (ext && ['mov', 'mp4'].includes(ext)) {
videos.push(obj)
}
}
// 匹配 Live Photo
for (const photo of photos) {
const photoKey = 'Key' in photo ? photo.Key : (photo as StorageObject).key
if (!photoKey) continue
const photoBaseName = photoKey.replace(/\.[^/.]+$/, '')
// 查找对应的视频文件
const matchingVideo = videos.find((video) => {
const videoKey = 'Key' in video ? video.Key : (video as StorageObject).key
if (!videoKey) return false
const videoBaseName = videoKey.replace(/\.[^/.]+$/, '')
return videoBaseName === photoBaseName
})
if (matchingVideo) {
livePhotoMap.set(photoKey, matchingVideo)
}
}
return livePhotoMap
}

View File

@@ -0,0 +1,145 @@
import type { ConsolaInstance } from 'consola'
import type { Logger, WorkerLogger } from '../logger/index.js'
/**
* 通用 Logger 接口
*/
export interface PhotoLogger {
info: (message: string, ...args: any[]) => void
warn: (message: string, ...args: any[]) => void
error: (message: string, error?: any) => void
success: (message: string, ...args: any[]) => void
}
/**
* Logger 适配器基类
*/
export abstract class LoggerAdapter implements PhotoLogger {
abstract info(message: string, ...args: any[]): void
abstract warn(message: string, ...args: any[]): void
abstract error(message: string, error?: any): void
abstract success(message: string, ...args: any[]): void
}
/**
* Worker Logger 适配器
* 将现有的 Logger 系统适配到通用接口
*/
export class WorkerLoggerAdapter extends LoggerAdapter {
constructor(private logger: ReturnType<WorkerLogger['withTag']>) {
super()
}
info(message: string, ...args: any[]): void {
this.logger.info(message, ...args)
}
warn(message: string, ...args: any[]): void {
this.logger.warn(message, ...args)
}
error(message: string, error?: any): void {
this.logger.error(message, error)
}
success(message: string, ...args: any[]): void {
this.logger.success(message, ...args)
}
// 提供原始 logger 实例,用于需要 ConsolaInstance 的地方
get originalLogger(): ConsolaInstance {
return this.logger
}
}
/**
* 兼容的 Logger 适配器
* 既实现 PhotoLogger 接口,又提供原始 ConsolaInstance
*/
export class CompatibleLoggerAdapter implements PhotoLogger {
private logger: ReturnType<WorkerLogger['withTag']>
constructor(logger: ReturnType<WorkerLogger['withTag']>) {
this.logger = logger
// 将原始 logger 的所有属性和方法复制到当前实例
Object.setPrototypeOf(this, logger)
Object.assign(this, logger)
}
info(message: string, ...args: any[]): void {
this.logger.info(message, ...args)
}
warn(message: string, ...args: any[]): void {
this.logger.warn(message, ...args)
}
error(message: string, error?: any): void {
this.logger.error(message, error)
}
success(message: string, ...args: any[]): void {
this.logger.success(message, ...args)
}
// 提供原始 logger 实例
get originalLogger(): ConsolaInstance {
return this.logger
}
}
/**
* 照片处理专用的 Logger 集合
*/
export interface PhotoProcessingLoggers {
image: CompatibleLoggerAdapter
s3: CompatibleLoggerAdapter
thumbnail: CompatibleLoggerAdapter
blurhash: CompatibleLoggerAdapter
exif: CompatibleLoggerAdapter
tone: CompatibleLoggerAdapter
}
/**
* 创建照片处理 Logger 集合
*/
export function createPhotoProcessingLoggers(
workerId: number,
baseLogger: Logger,
): PhotoProcessingLoggers {
const workerLogger = baseLogger.worker(workerId)
return {
image: new CompatibleLoggerAdapter(workerLogger.withTag('IMAGE')),
s3: new CompatibleLoggerAdapter(workerLogger.withTag('S3')),
thumbnail: new CompatibleLoggerAdapter(workerLogger.withTag('THUMBNAIL')),
blurhash: new CompatibleLoggerAdapter(workerLogger.withTag('BLURHASH')),
exif: new CompatibleLoggerAdapter(workerLogger.withTag('EXIF')),
tone: new CompatibleLoggerAdapter(workerLogger.withTag('TONE')),
}
}
/**
* 全局 Logger 实例
*/
let globalLoggers: PhotoProcessingLoggers | null = null
/**
* 设置全局 Logger
*/
export function setGlobalLoggers(loggers: PhotoProcessingLoggers): void {
globalLoggers = loggers
}
/**
* 获取全局 Logger
*/
export function getGlobalLoggers(): PhotoProcessingLoggers {
if (!globalLoggers) {
throw new Error(
'Global loggers not initialized. Call setGlobalLoggers first.',
)
}
return globalLoggers
}

View File

@@ -1,32 +1,15 @@
import path from 'node:path'
import { workdir } from '@afilmory/builder/path.js'
import type { _Object } from '@aws-sdk/client-s3'
import sharp from 'sharp'
import { HEIC_FORMATS } from '../constants/index.js'
import { extractExifData } from '../image/exif.js'
import { calculateHistogramAndAnalyzeTone } from '../image/histogram.js'
import {
convertBmpToJpegSharpInstance,
getImageMetadataWithSharp,
isBitmap,
preprocessImageBuffer,
} from '../image/processor.js'
import {
generateThumbnailAndBlurhash,
thumbnailExists,
} from '../image/thumbnail.js'
import type { Logger } from '../logger/index.js'
import { logger } from '../logger/index.js'
import { needsUpdate } from '../manifest/manager.js'
import { defaultStorageManager } from '../storage/manager.js'
import type {
PhotoManifestItem,
PickedExif,
ProcessPhotoResult,
} from '../types/photo.js'
import { extractPhotoInfo } from './info-extractor.js'
import type { PhotoManifestItem, ProcessPhotoResult } from '../types/photo.js'
import type { PhotoProcessingContext } from './image-pipeline.js'
import { processPhotoWithPipeline } from './image-pipeline.js'
import {
createPhotoProcessingLoggers,
setGlobalLoggers,
} from './logger-adapter.js'
export interface PhotoProcessorOptions {
isForceMode: boolean
@@ -34,15 +17,6 @@ export interface PhotoProcessorOptions {
isForceThumbnails: boolean
}
export interface WorkerLoggers {
image: Logger['image']
s3: Logger['s3']
thumbnail: Logger['thumbnail']
blurhash: Logger['blurhash']
exif: Logger['exif']
tone: Logger['image'] // 影调分析使用 image logger
}
// 处理单张照片
export async function processPhoto(
obj: _Object,
@@ -62,232 +36,22 @@ export async function processPhoto(
const photoId = path.basename(key, path.extname(key))
const existingItem = existingManifestMap.get(key)
// 创建 worker 专用的 logger
const workerLoggers: WorkerLoggers = {
image: logger.worker(workerId).withTag('IMAGE'),
s3: logger.worker(workerId).withTag('S3'),
thumbnail: logger.worker(workerId).withTag('THUMBNAIL'),
blurhash: logger.worker(workerId).withTag('BLURHASH'),
exif: logger.worker(workerId).withTag('EXIF'),
tone: logger.worker(workerId).withTag('TONE'),
// 创建并设置全局 logger
const photoLoggers = createPhotoProcessingLoggers(workerId, logger)
setGlobalLoggers(photoLoggers)
photoLoggers.image.info(`📸 [${index + 1}/${totalImages}] ${key}`)
// 构建处理上下文
const context: PhotoProcessingContext = {
photoKey: key,
photoId,
obj,
existingItem,
livePhotoMap,
options,
}
workerLoggers.image.info(`📸 [${index + 1}/${totalImages}] ${key}`)
// 检查是否需要更新
if (
!options.isForceMode &&
!options.isForceManifest &&
existingItem &&
!needsUpdate(existingItem, obj)
) {
// 检查缩略图是否存在,如果不存在或强制刷新缩略图则需要重新处理
const hasThumbnail = await thumbnailExists(photoId)
if (hasThumbnail && !options.isForceThumbnails) {
workerLoggers.image.info(`⏭️ 跳过处理 (未更新且缩略图存在): ${key}`)
return { item: existingItem, type: 'skipped' }
} else {
if (options.isForceThumbnails) {
workerLoggers.image.info(`🔄 强制重新生成缩略图:${key}`)
} else {
workerLoggers.image.info(
`🔄 重新生成缩略图 (文件未更新但缩略图缺失): ${key}`,
)
}
}
}
// 需要处理的照片(新照片、更新的照片或缺失缩略图的照片)
const isNewPhoto = !existingItem
if (isNewPhoto) {
workerLoggers.image.info(`🆕 新照片:${key}`)
} else {
workerLoggers.image.info(`🔄 更新照片:${key}`)
}
try {
// 获取图片数据
const rawImageBuffer = await defaultStorageManager.getFile(key)
if (!rawImageBuffer) return { item: null, type: 'failed' }
// 预处理图片(处理 HEIC/HEIF 格式)
let imageBuffer: Buffer
try {
imageBuffer = await preprocessImageBuffer(
rawImageBuffer,
key,
workerLoggers.image,
)
} catch (error) {
workerLoggers.image.error(`预处理图片失败:${key}`, error)
return { item: null, type: 'failed' }
}
// 创建 Sharp 实例,复用于多个操作
let sharpInstance = sharp(imageBuffer)
// 处理 BMP
if (isBitmap(imageBuffer)) {
try {
// Convert the BMP image to JPEG format and create a new Sharp instance for the converted image.
sharpInstance = await convertBmpToJpegSharpInstance(
imageBuffer,
workerLoggers.image,
)
// Update the image buffer to reflect the new JPEG data from the Sharp instance.
imageBuffer = await sharpInstance.toBuffer()
} catch (error) {
workerLoggers.image.error(`转换 BMP 失败:${key}`, error)
return { item: null, type: 'failed' }
}
}
// 获取图片元数据(复用 Sharp 实例)
const metadata = await getImageMetadataWithSharp(
sharpInstance,
workerLoggers.image,
)
if (!metadata) return { item: null, type: 'failed' }
// 如果是增量更新且已有 blurhash可以复用
let thumbnailUrl: string | null = null
let thumbnailBuffer: Buffer | null = null
let blurhash: string | null = null
if (
!options.isForceMode &&
!options.isForceThumbnails &&
existingItem?.blurhash &&
(await thumbnailExists(photoId))
) {
// 复用现有的缩略图和 blurhash
blurhash = existingItem.blurhash
workerLoggers.blurhash.info(`复用现有 blurhash: ${photoId}`)
try {
const fs = await import('node:fs/promises')
const thumbnailPath = path.join(
workdir,
'public/thumbnails',
`${photoId}.webp`,
)
thumbnailBuffer = await fs.readFile(thumbnailPath)
thumbnailUrl = `/thumbnails/${photoId}.webp`
workerLoggers.thumbnail.info(`复用现有缩略图:${photoId}`)
} catch (error) {
workerLoggers.thumbnail.warn(
`读取现有缩略图失败,重新生成:${photoId}`,
error,
)
// 继续执行生成逻辑
}
}
// 如果没有复用成功,则生成缩略图和 blurhash
if (!thumbnailUrl || !thumbnailBuffer || !blurhash) {
const result = await generateThumbnailAndBlurhash(
imageBuffer,
photoId,
metadata.width,
metadata.height,
options.isForceMode || options.isForceThumbnails,
{
thumbnail: workerLoggers.thumbnail,
blurhash: workerLoggers.blurhash,
},
)
thumbnailUrl = result.thumbnailUrl
thumbnailBuffer = result.thumbnailBuffer
blurhash = result.blurhash
}
// 如果是增量更新且已有 EXIF 数据,可以复用
let exifData: PickedExif | null = null
if (
!options.isForceMode &&
!options.isForceManifest &&
existingItem?.exif
) {
exifData = existingItem.exif
workerLoggers.exif.info(`复用现有 EXIF 数据:${photoId}`)
} else {
// 传入原始 buffer 以便在转换后的图片缺少 EXIF 时回退
const ext = path.extname(key).toLowerCase()
const originalBuffer = HEIC_FORMATS.has(ext) ? rawImageBuffer : undefined
exifData = await extractExifData(
imageBuffer,
originalBuffer,
workerLoggers.exif,
)
}
// 计算影调分析
let toneAnalysis: import('../types/photo.js').ToneAnalysis | null = null
if (
!options.isForceMode &&
!options.isForceManifest &&
existingItem?.toneAnalysis
) {
toneAnalysis = existingItem.toneAnalysis
workerLoggers.tone.info(`复用现有影调分析:${photoId}`)
} else {
toneAnalysis = await calculateHistogramAndAnalyzeTone(
sharpInstance,
workerLoggers.tone,
)
}
// 提取照片信息(在获取 EXIF 数据之后,以便使用 DateTimeOriginal
const photoInfo = extractPhotoInfo(key, exifData, workerLoggers.image)
const aspectRatio = metadata.width / metadata.height
// 检查是否为 live photo
const livePhotoVideo = livePhotoMap.get(key)
const isLivePhoto = !!livePhotoVideo
let livePhotoVideoUrl: string | undefined
let livePhotoVideoS3Key: string | undefined
if (isLivePhoto && livePhotoVideo?.Key) {
livePhotoVideoS3Key = livePhotoVideo.Key
livePhotoVideoUrl = defaultStorageManager.generatePublicUrl(
livePhotoVideo.Key,
)
workerLoggers.image.info(
`📱 检测到 Live Photo${key} -> ${livePhotoVideo.Key}`,
)
}
const photoItem: PhotoManifestItem = {
id: photoId,
title: photoInfo.title,
description: photoInfo.description,
dateTaken: photoInfo.dateTaken,
views: photoInfo.views,
tags: photoInfo.tags,
originalUrl: defaultStorageManager.generatePublicUrl(key),
thumbnailUrl,
blurhash,
width: metadata.width,
height: metadata.height,
aspectRatio,
s3Key: key,
lastModified: obj.LastModified?.toISOString() || new Date().toISOString(),
size: obj.Size || 0,
exif: exifData,
toneAnalysis,
// Live Photo 相关字段
isLivePhoto,
livePhotoVideoUrl,
livePhotoVideoS3Key,
}
workerLoggers.image.success(`✅ 处理完成:${key}`)
return { item: photoItem, type: isNewPhoto ? 'new' : 'processed' }
} catch (error) {
workerLoggers.image.error(`❌ 处理失败:${key}`, error)
return { item: null, type: 'failed' }
}
// 使用处理管道
return await processPhotoWithPipeline(context)
}