---
title: Storage providers
description: Afilmory can work with multiple storage providers, including S3, Git, Eagle, and local file system
createdAt: 2025-08-12T15:09:08+08:00
lastModified: 2025-10-23T18:31:41+08:00
---

# Storage Providers

Afilmory's flexible storage architecture allows you to store your photos across different platforms. The photo processing engine (`@afilmory/builder`) abstracts storage operations through a unified interface, making it easy to switch between providers such as S3, GitHub, Eagle, and the local file system.

## Supported Providers

### S3-Compatible Storage

S3-compatible storage is the recommended option for production deployments, offering scalability, CDN integration, and reliable performance.

**Configuration in `builder.config.json`:**

```json
{
  "storage": {
    "provider": "s3",
    "bucket": "your-photos-bucket",
    "region": "us-east-1",
    "prefix": "photos/",
    "customDomain": "cdn.yourdomain.com"
  }
}
```

**Environment Variables (`.env`):**

```bash
S3_ACCESS_KEY_ID=your_access_key_id
S3_SECRET_ACCESS_KEY=your_secret_access_key
```

### GitHub Storage

GitHub storage leverages Git repositories for photo storage, ideal for static sites, small galleries, or when you want version control for your photos.

**Features:**
- ✅ Free storage space (GitHub repository limit 1GB)
- ✅ Global CDN support via raw.githubusercontent.com
- ✅ Version control for your photos
- ✅ Public access without authentication
- ⚠️ GitHub API rate limits apply
- ⚠️ Not suitable for large files or frequent updates

**Configuration in `builder.config.json`:**

```json
{
  "storage": {
    "provider": "github",
    "owner": "your-username",
    "repo": "photo-storage",
    "branch": "main",
    "path": "photos",
    "useRawUrl": true
  }
}
```

**Environment Variables:**

```bash
GIT_TOKEN=ghp_your_github_personal_access_token
```

**Setup Steps:**

1. **Create GitHub Repository**
   ```bash
   git clone https://github.com/your-username/photo-gallery.git
   cd photo-gallery
   mkdir photos
   ```

2. **Get GitHub Access Token** (optional but recommended)
   - Visit GitHub Settings > Developer settings > Personal access tokens
   - Create new Fine-grained personal access token
   - Grant "Contents" permission (read/write) to your repository

3. **Required Permissions:**
   - `Contents: Read and write` - for uploading processed photos
   - `Metadata: Read` - for repository access

### Eagle Storage

Eagle storage integrates directly with an existing Eagle 4 desktop library, making it ideal for teams that already curate assets in Eagle and want to publish selected photos without additional uploads.

**Features:**
- ✅ No upload step: reads directly from the Eagle library
- ✅ Flexible filtering: include or exclude by folder (with optional subfolders) and tags
- ✅ Automatic publishing: copies originals to a public directory on demand
- ⚠️ Requires absolute paths for `libraryPath` and `distPath`
- ⚠️ Only Eagle 4.x is validated; other versions may require testing

**Configuration in `builder.config.json`:**

```json
{
  "storage": {
    "provider": "eagle",
    "libraryPath": "/Users/alice/Pictures/Eagle.library",
    "distPath": "/Users/alice/workspaces/afilmory/apps/web/public/originals",
    "baseUrl": "/originals/",
    "include": [
      { "type": "folder", "name": "Published", "includeSubfolder": true },
      { "type": "tag", "name": "Featured" }
    ],
    "exclude": [
      { "type": "tag", "name": "Private" }
    ]
  }
}
```

**Setup Steps:**

1. **Locate the Eagle library**: copy the absolute path shown in Eagle → Preferences → Library.
2. **Choose a distribution path (optional)**: The original images will be copied to `distPath` during the build process (defaults to `<workdir>/public/originals`).
3. **Align URLs**: set `baseUrl` to match how the frontend serves `distPath` (defaults to `/originals/`).
4. **Refine filters (optional)**: use `include` and `exclude` rules to control which assets are exported.

**Rule Notes:**
- Leaving `include` empty processes every supported image in the library.
- `exclude` overrides inclusion rules when both match a single asset.
- `folder` rules use folder names (not full paths); enabling `includeSubfolder` recursively matches children.
- `tag` rules are case-sensitive and match Eagle tags.

### Local File System

Local storage is suitable for development, testing, or self-hosted deployments where photos are stored on the same server.

**Features:**
- ✅ No external dependencies
- ✅ Fast access speeds
- ✅ Complete private control
- ✅ Recursive directory scanning
- ✅ Live Photos detection support
- ⚠️ Requires proper file system permissions
- ⚠️ Not suitable for distributed deployments

**Configuration in `builder.config.json`:**

Self-hosted original image (manually serve to `basePath`):

```json
{
  "storage": {
    "provider": "local",
    "basePath": "./photos",
    "baseUrl": "http://example.com:3000/photos/"
  }
}
```

Hosted with frontend (serve via app static assets):

```json
{
  "storage": {
    "provider": "local",
    "basePath": "./photos",
    "distPath": "./apps/web/public/originals",
    "baseUrl": "/originals/"
  }
}
```

  Use when you want originals deployed alongside your frontend’s static assets. On initialization/build, files from `basePath` are copied to `distPath` (overwriting same-named files).

## Photo Processing Workflow

### Image Processing Pipeline

When photos are processed, Afilmory performs these operations:

1. **Format Detection** - Identifies HEIC, TIFF, RAW, and standard formats
2. **EXIF Extraction** - Extracts camera metadata, GPS data, and Fujifilm film recipes
3. **Format Conversion** - Converts HEIC/TIFF to web-compatible formats (WebP, JPEG)
4. **Thumbnail Generation** - Creates multiple thumbnail sizes
5. **Blurhash Generation** - Creates placeholder images for smooth loading
6. **Manifest Creation** - Generates metadata JSON for the gallery

### Storage Organization

Afilmory organizes files in your storage with this structure:

```
photos/
├── 2024/
│   ├── 01-january/
│   │   ├── IMG_001.jpg
│   │   ├── IMG_001.mov      # Live Photo video
│   │   └── IMG_002.heic
│   └── 02-february/
│       └── sunset.jpg
├── 2023/
│   └── vacation/
│       ├── beach.jpg
│       └── mountain.png
└── manifest.json            # Photo metadata and index
```

### Processing Commands

Use these commands to manage your photo processing:

```bash
# Initial photo processing
pnpm run build:manifest

# Force rebuild all photos
pnpm run build:manifest -- --force

# Regenerate only thumbnails
pnpm run build:manifest -- --force-thumbnails

# Process specific directory
pnpm run build:manifest -- --path "2024/vacation"
```

## Environment-Specific Configurations

### Development Setup

For local development, use local storage:

```json
{
  "storage": {
    "provider": "local",
    "basePath": "./dev-photos",
    "baseUrl": "http://localhost:1924/photos"
  }
}
```

### Production Setup

For production, use S3-compatible storage with CDN:

```json
{
  "storage": {
    "provider": "s3",
    "bucket": "prod-photos-bucket",
    "region": "us-east-1",
    "prefix": "photos/",
    "customDomain": "cdn.yourdomain.com"
  },
  "performance": {
    "worker": {
      "enabled": true,
      "maxWorkers": 4
    }
  }
}
```

## API Limitations

### GitHub Storage Limits

- **Unauthenticated requests**: 60 requests/hour/IP
- **Authenticated requests**: 5,000 requests/hour/token
- **File size**: Maximum 100MB (via API)
- **Repository size**: Recommended not to exceed 1GB

### Storage Provider Comparison

| Feature | S3 | GitHub | Local |
|---------|----|----|--------|
| Storage Space | Pay-as-you-go | 1GB free | Depends on disk |
| CDN | Additional cost | Free global CDN | Manual setup |
| API Limits | Very high | Limited | None |
| Use Case | Production | Small projects, demos | Development, self-hosted |
| Setup Complexity | Medium | Simple | Minimal |

## Security Considerations

### S3 Bucket Policies

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::your-bucket/photos/*"
    }
  ]
}
```

### GitHub Repository Settings

- Set repository to public for image access
- Use fine-grained personal access tokens
- Limit token permissions to specific repositories

### Environment Variables

Store sensitive credentials securely:

```bash
# Use environment-specific .env files
# .env.production
S3_ACCESS_KEY_ID=prod_access_key
S3_SECRET_ACCESS_KEY=prod_secret_key

# .env.development
S3_ACCESS_KEY_ID=dev_access_key
S3_SECRET_ACCESS_KEY=dev_secret_key
```