<!-- Generated by SpecStory -->

# 根据图片处理开启 cluster (2025-06-15 13:15:10)

_**User**_

这里需要根据实际要处理的图片再去开启 cluster。例如不是 force 模式下，如果新增一张图片，其实只有一个任务，不需要开启这么多 cluster 去做

---

_**Assistant**_

我来分析代码，看看如何优化cluster的启用逻辑。首先让我查看相关的代码来理解当前的实现。

---

Read file: packages/builder/src/builder/builder.ts

---

现在让我查看一下如何确定实际需要处理的图片数量。当前的逻辑是基于`imageObjects.length`来决定，但实际上需要处理的可能只是其中一部分（新增的或者需要更新的）。让我看看`processPhoto`函数的实现：

---

Read file: packages/builder/src/photo/processor.ts

---

<details>
            <summary>Grep search for "needsUpdate" • **3** files</summary>
        
| File | Line | Match |
|------|------|-------|
| `file:///Users/innei/git/vibe-coding/photo-gallery-site/packages/builder/src/photo/processor.ts` | L19 | `import { needsUpdate } from '../manifest/manager.js'` |
| `file:///Users/innei/git/vibe-coding/photo-gallery-site/packages/builder/src/photo/processor.ts` | L78 | `!needsUpdate(existingItem, obj)` |
| `file:///Users/innei/git/vibe-coding/photo-gallery-site/packages/builder/src/index.ts` | L51 | `needsUpdate,` |
| `file:///Users/innei/git/vibe-coding/photo-gallery-site/packages/builder/src/manifest/manager.ts` | L21 | `export function needsUpdate(` |

</details>

---

Read file: packages/builder/src/manifest/manager.ts

---

现在我明白了问题。当前的逻辑是基于`imageObjects.length`来决定是否使用cluster模式，但实际上需要处理的图片可能要少得多。在非force模式下，很多图片可能会被跳过（因为它们已经是最新的并且缩略图存在）。

我需要实现一个预检查逻辑，在决定使用cluster模式之前，先计算出实际需要处理的图片数量。让我修改代码：

---

```diff
  
        if (imageObjects.length > 0) {
+         // 预检查：计算实际需要处理的图片数量
+         const actualTasksCount = await this.calculateActualTasks(
+           imageObjects,
+           existingManifestMap,
+           options,
+         )
+ 
+         logger.main.info(`存储中找到 ${imageObjects.length} 张照片，实际需要处理 ${actualTasksCount} 张`)
+ 
          // 获取并发限制
          const concurrency =
            options.concurrencyLimit ?? this.config.options.defaultConcurrency
  
-         // 根据配置选择处理模式
+         // 根据配置和实际任务数量选择处理模式
          const { useClusterMode } = this.config.performance.worker
+         
+         // 如果实际任务数量较少，则不使用cluster模式
+         const shouldUseCluster = useClusterMode && actualTasksCount >= concurrency * 2
  
          logger.main.info(
-           `开始${useClusterMode ? '多进程' : '并发'}处理任务，${useClusterMode ? '进程' : 'Worker'}数：${concurrency}${useClusterMode ? `，每进程并发：${this.config.performance.worker.workerConcurrency}` : ''}`,
+           `开始${shouldUseCluster ? '多进程' : '并发'}处理任务，${shouldUseCluster ? '进程' : 'Worker'}数：${concurrency}${shouldUseCluster ? `，每进程并发：${this.config.performance.worker.workerConcurrency}` : ''}`,
          )
  
```

```diff
          let results: ProcessPhotoResult[]
  
-         if (useClusterMode) {
+         if (shouldUseCluster) {
            // 创建 Cluster 池（多进程模式）
            const clusterPool = new ClusterPool<ProcessPhotoResult>(
```

---

我需要实现`calculateActualTasks`方法。让我添加这个方法：

---

Read file: packages/builder/src/image/thumbnail.ts

---

现在我需要实现`calculateActualTasks`方法。这个方法需要检查每张图片是否需要处理：

---

```diff
    loadExistingManifest,
    saveManifest,
+   needsUpdate,
  } from '../manifest/manager.js'
  import type { PhotoProcessorOptions } from '../photo/processor.js'
```

```diff
  import { ClusterPool } from '../worker/cluster-pool.js'
  import { WorkerPool } from '../worker/pool.js'
+ import { thumbnailExists } from '../image/thumbnail.js'
+ import path from 'node:path'
  
  export interface BuilderOptions {
```

```diff
      return { ...this.config }
    }
+ 
+   private async calculateActualTasks(
+     imageObjects: (typeof imageObjects)[0][],
+     existingManifestMap: Map<string, PhotoManifestItem>,
+     options: BuilderOptions,
+   ): Promise<number> {
+     const actualTasks = new Set<string>()
+ 
+     for (const obj of imageObjects) {
+       const key = obj.key
+       if (existingManifestMap.has(key)) {
+         actualTasks.add(key)
+       } else if (!options.isForceMode && !options.isForceManifest) {
+         actualTasks.add(key)
+       }
+     }
+ 
+     return actualTasks.size
+   }
  }
  
```

---

我需要修复类型注解问题并正确实现`calculateActualTasks`方法：

---

```diff
    }
  
+   /**
+    * 计算实际需要处理的图片数量
+    * @param imageObjects 存储中的图片对象列表
+    * @param existingManifestMap 现有manifest的映射
+    * @param options 构建选项
+    * @returns 实际需要处理的图片数量
+    */
    private async calculateActualTasks(
-     imageObjects: (typeof imageObjects)[0][],
+     imageObjects: Awaited<ReturnType<StorageManager['listImages']>>,
      existingManifestMap: Map<string, PhotoManifestItem>,
      options: BuilderOptions,
    ): Promise<number> {
-     const actualTasks = new Set<string>()
+     let actualTasksCount = 0
  
      for (const obj of imageObjects) {
-       const { key } = obj
-       if (existingManifestMap.has(key)) {
-         actualTasks.add(key)
-       } else if (!options.isForceMode && !options.isForceManifest) {
-         actualTasks.add(key)
+       const key = obj.key
+       const photoId = path.basename(key, path.extname(key))
+       const existingItem = existingManifestMap.get(key)
+ 
+       // 强制模式下所有图片都需要处理
+       if (options.isForceMode || options.isForceManifest) {
+         actualTasksCount++
+         continue
+       }
+ 
+       // 新图片需要处理
+       if (!existingItem) {
+         actualTasksCount++
+         continue
+       }
+ 
+       // 检查是否需要更新（基于修改时间）
+       const legacyObj = {
+         Key: key,
+         Size: obj.size,
+         LastModified: obj.lastModified,
+         ETag: obj.etag,
+       }
+       
+       if (needsUpdate(existingItem, legacyObj)) {
+         actualTasksCount++
+         continue
+       }
+ 
+       // 检查缩略图是否存在，如果不存在或强制刷新缩略图则需要处理
+       const hasThumbnail = await thumbnailExists(photoId)
+       if (!hasThumbnail || options.isForceThumbnails) {
+         actualTasksCount++
+         continue
        }
+ 
+       // 其他情况下跳过处理
      }
  
-     return actualTasks.size
+     return actualTasksCount
    }
  }
```

---

现在让我检查一下代码是否有错误：

---

```bash
cd packages/builder && npm run type-check
```

```
 ERR_PNPM_NO_SCRIPT  Missing script: type-check

Command "type-check" not found.

```

---

```bash
npx tsc --noEmit
```

```
at 14:03:14 ❯ npx tsc --noEmit

                                                                               
                This is not the tsc command you are looking for                
                                                                               

To get access to the TypeScript compiler, tsc, from the command line either:

- Use npm install typescript to first add TypeScript to your project before using npx
- Use yarn to avoid accidentally running code from un-installed packages

innei in photo-gallery-site/packages/builder on  main [$!] via  v22.15.0 on ☁️  innei@rss3.io at     54312be "re
factor: remove image preloading logic from PhotoViewer component"  by tukon479@gmail.com 
at 14:03:21 ❯
```

---

```bash
cd ../../ && npm run build:builder
```

```
at 14:03:21 ❯ cd ../../ && npm run build:builder
 ERR_PNPM_NO_SCRIPT  Missing script: build:builder

Command "build:builder" not found. Did you mean "pnpm run build:manifest"?

innei in photo-gallery-site on  main [$!] via 🐳 orbstack via  v22.15.0 on ☁️  innei@rss3.io at     54312be "ref
actor: remove image preloading logic from PhotoViewer component"  by tukon479@gmail.com 
at 14:03:26 ❯
```

---

_**User**_

需要计算出需要处理的图片的整个 array 替换 imageObjects 直接当做 taskdata

---

_**Assistant**_

我明白了，您希望不仅计算数量，而是要筛选出实际需要处理的图片数组，然后用这个筛选后的数组替换原来的`imageObjects`作为任务数据。让我修改代码实现这个功能。

---

```diff
  
        if (imageObjects.length > 0) {
-         // 预检查：计算实际需要处理的图片数量
-         const actualTasksCount = await this.calculateActualTasks(
+         // 筛选出实际需要处理的图片
+         const tasksToProcess = await this.filterTaskImages(
            imageObjects,
            existingManifestMap,
            options,
          )
  
-         logger.main.info(
-           `存储中找到 ${imageObjects.length} 张照片，实际需要处理 ${actualTasksCount} 张`,
-         )
+         logger.main.info(`存储中找到 ${imageObjects.length} 张照片，实际需要处理 ${tasksToProcess.length} 张`)
  
          // 获取并发限制
```

```diff
          
          // 如果实际任务数量较少，则不使用cluster模式
-         const shouldUseCluster =
-           useClusterMode && actualTasksCount >= concurrency * 2
+         const shouldUseCluster = useClusterMode && tasksToProcess.length >= concurrency * 2
  
          logger.main.info(
```

```diff
              {
                concurrency,
-               totalTasks: imageObjects.length,
+               totalTasks: tasksToProcess.length,
                workerConcurrency:
                  this.config.performance.worker.workerConcurrency,
```

```diff
              {
                concurrency,
-               totalTasks: imageObjects.length,
+               totalTasks: tasksToProcess.length,
              },
              logger,
```

```diff
            // 执行并发处理
            results = await workerPool.execute(async (taskIndex, workerId) => {
-             const obj = imageObjects[taskIndex]
+             const obj = tasksToProcess[taskIndex]
  
              // 转换 StorageObject 到旧的 _Object 格式以兼容现有的 processPhoto 函数
```

```diff
                taskIndex,
                workerId,
-               imageObjects.length,
+               tasksToProcess.length,
                existingManifestMap,
                legacyLivePhotoMap,
```

```diff
      return actualTasksCount
    }
+ 
+   private async filterTaskImages(
+     imageObjects: Awaited<ReturnType<StorageManager['listImages']>>,
+     existingManifestMap: Map<string, PhotoManifestItem>,
+     options: BuilderOptions,
+   ): Promise<Awaited<ReturnType<StorageManager['listImages']>>> {
+     const tasksToProcess: Awaited<ReturnType<StorageManager['listImages']>> = []
+ 
+     for (const obj of imageObjects) {
+       const { key } = obj
+       const photoId = path.basename(key, path.extname(key))
+       const existingItem = existingManifestMap.get(key)
+ 
+       // 新图片需要处理
+       if (!existingItem) {
+         tasksToProcess.push(obj)
+         continue
+       }
+ 
+       // 检查是否需要更新（基于修改时间）
+       const legacyObj = {
+         Key: key,
+         Size: obj.size,
+         LastModified: obj.lastModified,
+         ETag: obj.etag,
+       }
+ 
+       if (needsUpdate(existingItem, legacyObj)) {
+         tasksToProcess.push(obj)
+         continue
+       }
+ 
+       // 检查缩略图是否存在，如果不存在或强制刷新缩略图则需要处理
+       const hasThumbnail = await thumbnailExists(photoId)
+       if (!hasThumbnail || options.isForceThumbnails) {
+         tasksToProcess.push(obj)
+         continue
+       }
+ 
+       // 其他情况下跳过处理
+     }
+ 
+     return tasksToProcess
+   }
  }
  
```

---

现在我需要更新原有的`calculateActualTasks`方法名称和在`filterTaskImages`中处理强制模式的逻辑：

---

```diff
  
    /**
-    * 计算实际需要处理的图片数量
+    * 筛选出实际需要处理的图片
     * @param imageObjects 存储中的图片对象列表
     * @param existingManifestMap 现有manifest的映射
     * @param options 构建选项
-    * @returns 实际需要处理的图片数量
+    * @returns 实际需要处理的图片数组
     */
-   private async calculateActualTasks(
-     imageObjects: Awaited<ReturnType<StorageManager['listImages']>>,
-     existingManifestMap: Map<string, PhotoManifestItem>,
-     options: BuilderOptions,
-   ): Promise<number> {
-     if (options.isForceMode || options.isForceManifest) {
-       return imageObjects.length
-     }
- 
-     let actualTasksCount = 0
- 
-     for (const obj of imageObjects) {
-       const { key } = obj
-       const photoId = path.basename(key, path.extname(key))
-       const existingItem = existingManifestMap.get(key)
- 
-       // 新图片需要处理
-       if (!existingItem) {
-         actualTasksCount++
-         continue
-       }
- 
-       // 检查是否需要更新（基于修改时间）
-       const legacyObj = {
-         Key: key,
-         Size: obj.size,
-         LastModified: obj.lastModified,
-         ETag: obj.etag,
-       }
- 
-       if (needsUpdate(existingItem, legacyObj)) {
-         actualTasksCount++
-         continue
-       }
- 
-       // 检查缩略图是否存在，如果不存在或强制刷新缩略图则需要处理
-       const hasThumbnail = await thumbnailExists(photoId)
-       if (!hasThumbnail || options.isForceThumbnails) {
-         actualTasksCount++
-         continue
-       }
- 
-       // 其他情况下跳过处理
-     }
- 
-     return actualTasksCount
-   }
- 
    private async filterTaskImages(
      imageObjects: Awaited<ReturnType<StorageManager['listImages']>>,
      existingManifestMap: Map<string, PhotoManifestItem>,
      options: BuilderOptions,
    ): Promise<Awaited<ReturnType<StorageManager['listImages']>>> {
+     // 强制模式下所有图片都需要处理
+     if (options.isForceMode || options.isForceManifest) {
+       return imageObjects
+     }
+ 
      const tasksToProcess: Awaited<ReturnType<StorageManager['listImages']>> = []
  
      for (const obj of imageObjects) {
-       const { key } = obj
+       const key = obj.key
        const photoId = path.basename(key, path.extname(key))
        const existingItem = existingManifestMap.get(key)
```

---

现在让我检查一下修改后的代码。我还需要确保在日志中正确显示需要处理的任务数量，以及在没有任务时跳过处理：

---

Read file: packages/builder/src/builder/builder.ts

---

我注意到还需要处理一个情况：当筛选后没有任务需要处理时，应该跳过处理逻辑但仍然需要处理已存在的manifest项目。让我添加这个逻辑：

---

```diff
          )
  
-         logger.main.info(
-           `存储中找到 ${imageObjects.length} 张照片，实际需要处理 ${tasksToProcess.length} 张`,
-         )
+         logger.main.info(`存储中找到 ${imageObjects.length} 张照片，实际需要处理 ${tasksToProcess.length} 张`)
  
+         // 如果没有任务需要处理，直接使用现有的manifest
+         if (tasksToProcess.length === 0) {
+           logger.main.info('💡 没有需要处理的照片，使用现有manifest')
+           manifest.push(...existingManifest.filter(item => s3ImageKeys.has(item.s3Key)))
+         } else {
            // 获取并发限制
            const concurrency =
```

```diff
            
            // 如果实际任务数量较少，则不使用cluster模式
-         const shouldUseCluster =
-           useClusterMode && tasksToProcess.length >= concurrency * 2
+           const shouldUseCluster = useClusterMode && tasksToProcess.length >= concurrency * 2
  
            logger.main.info(
```

```diff
                    break
                  }
+               }
+             }
+           }
+ 
+           // 添加未处理但仍然存在的照片到manifest
+           for (const [key, item] of existingManifestMap) {
+             if (s3ImageKeys.has(key) && !manifest.some(m => m.s3Key === key)) {
+               manifest.push(item)
+               skippedCount++
              }
            }
```

---

现在让我验证一下代码的逻辑是否完整。让我查看完整的修改后的代码段：

---

Read file: packages/builder/src/builder/builder.ts

---

现在我发现日志消息有格式问题，让我修复一下：

---



---

让我测试一下修改后的功能是否正常工作：

---

```bash
cd packages/builder && npm run build
```

```
at 14:03:26 ❯ cd packages/builder && npm run build
 ERR_PNPM_NO_SCRIPT  Missing script: build

But script matched with build is present in the root of the workspace,
so you may run "pnpm -w run build"

innei in photo-gallery-site/packages/builder on  main [$!] via  v22.15.0 on ☁️  innei@rss3.io at     54312be "re
factor: remove image preloading logic from PhotoViewer component"  by tukon479@gmail.com 
at 14:08:07 ❯
```

---

