Deploy fails with 413 Payload Too Large - Implement R2 storage for audio files #8

New issue

Closed

opened 2025-11-01 22:53:35 +00:00 by navicore · 0 comments

navicore commented

2025-11-01 22:53:35 +00:00

(Migrated from github.com)

Problem

The current deploy publish implementation bundles audio files into the Cloudflare Pages deployment zip, causing a 413 Payload Too Large error. Cloudflare Pages Direct Upload API has a 25MB limit, which is insufficient for album releases with audio files.

Current (broken) flow:

Audio files → Bundled in static site zip → Upload to Pages → ❌ 413 Error

Root Cause

Audio files are included in the Pages deployment (crates/cli/src/commands/deploy.rs:350-375)
album.toml has an r2_bucket field, but R2 is not implemented (crates/deployer/src/cloudflare/mod.rs:2)
A 3-minute WAV file can be 30MB+, immediately exceeding the limit
Even compressed FLAC files for full albums will exceed 25MB

Solution

Implement proper separation using Cloudflare R2 for audio storage and Pages for static content.

Correct architecture:

Audio files → R2 bucket (unlimited size) → Public URLs via custom domain
Static site (HTML/CSS/JS) → Pages deployment (< 1MB)
Player → References R2 URLs for streaming

Domain Architecture

Primary domain (from album.toml): myalbum.example.com

Site: myalbum.example.com → Cloudflare Pages
Audio CDN: cdn.myalbum.example.com → R2 bucket

Benefits:

CORS: Same parent domain, no cross-origin issues
Clean teardown: Delete DNS records, R2 bucket, and Pages project together
Professional: No .pages.dev or .r2.dev in production URLs

Implementation Plan

1. R2 Bucket Management

Create R2 bucket on deploy publish
Bucket naming convention: {project-name}-audio
Enable public access via custom domain
Set appropriate CORS headers

2. Custom Domain for R2

Create DNS CNAME: cdn.{domain} → R2 bucket public URL
Configure R2 custom domain binding
Store DNS record IDs for cleanup

3. Audio Upload to R2

Upload audio files to R2 with S3-compatible API
Path structure: {project-name}/audio/track.flac
Set appropriate content-type headers (audio/flac, audio/wav, etc.)
Generate public URLs: https://cdn.{domain}/{project-name}/audio/track.flac

4. Generator Updates

Modify crates/generator to accept R2 URLs for audio
Update player.js template to use R2 URLs instead of relative paths
Ensure preview mode still works with local files

5. Split Deployment Flow

Upload audio files → R2 first
Build static site with R2 URLs embedded
Deploy HTML/CSS/JS → Pages (small, under limit)
Configure Pages custom domain

6. API-Driven Teardown (No Local State)

Derive everything from album.toml + API queries:

// Given album.toml + ~/.release-kit/config.toml
let project_name = derive_project_name(&album.artist.name, &album.metadata.title);
let bucket_name = format!("{}-audio", project_name);
let domain = &album.site.domain;

// Query Cloudflare API to find resources
1. Query Pages project by name → delete if exists
2. Query R2 bucket by name → delete all objects → delete bucket  
3. Query DNS records matching domain pattern → delete all
4. No orphaned resources!

Advantages:

✅ No stale state files
✅ Works across machines
✅ Self-healing (can always re-derive from album.toml)
✅ Matches existing status command pattern

7. Configuration Updates

Update deploy configure to document R2 permissions
Remove obsolete r2_bucket and pages_project fields from album.toml
Keep subdomain field for custom domain configuration

8. Helper Commands

Add deploy list to show all deployed projects
Detect orphaned resources (renamed albums)
Enhanced deploy status to show R2 bucket info

API Token Permissions

Update required permissions:

Account > Cloudflare Pages > Edit
Account > R2 > Edit
Zone > DNS > Edit
Zone > Zone > Read

Testing Checklist

Deploy album with small audio files (< 1MB each)
Deploy album with large WAV files (> 30MB each)
Verify audio streaming works from R2 URLs via custom domain
Verify preview mode still works locally
Test teardown removes Pages project, R2 bucket, and DNS records
Test cross-origin requests work correctly
Test deploy after renaming album in album.toml

Files to Modify

crates/cli/src/commands/deploy.rs - R2 upload logic, split deployment, DNS management
crates/deployer/src/cloudflare/mod.rs - R2 API client, DNS API client
crates/generator/src/lib.rs - Accept R2 URLs for audio
crates/cli/src/commands/build.rs - Pass audio URLs to generator
crates/core/src/types.rs - Update CloudflareConfig (remove obsolete fields)

Open Questions

Audio subdomain: Use cdn.{domain} or audio.{domain} or make it configurable?
Bucket lifecycle: One bucket per album (current plan) or shared bucket?
Partial failure handling: Retry logic if DNS creation succeeds but R2 upload fails?
Migration path: How to handle existing (broken) deployments?

This is a critical blocker - the deploy feature is non-functional without R2 support.

## Problem The current `deploy publish` implementation bundles audio files into the Cloudflare Pages deployment zip, causing a **413 Payload Too Large** error. Cloudflare Pages Direct Upload API has a 25MB limit, which is insufficient for album releases with audio files. **Current (broken) flow:** ``` Audio files → Bundled in static site zip → Upload to Pages → ❌ 413 Error ``` ## Root Cause - Audio files are included in the Pages deployment (crates/cli/src/commands/deploy.rs:350-375) - `album.toml` has an `r2_bucket` field, but R2 is not implemented (crates/deployer/src/cloudflare/mod.rs:2) - A 3-minute WAV file can be 30MB+, immediately exceeding the limit - Even compressed FLAC files for full albums will exceed 25MB ## Solution Implement proper separation using Cloudflare R2 for audio storage and Pages for static content. **Correct architecture:** ``` Audio files → R2 bucket (unlimited size) → Public URLs via custom domain Static site (HTML/CSS/JS) → Pages deployment (< 1MB) Player → References R2 URLs for streaming ``` ## Domain Architecture **Primary domain** (from `album.toml`): `myalbum.example.com` - **Site**: `myalbum.example.com` → Cloudflare Pages - **Audio CDN**: `cdn.myalbum.example.com` → R2 bucket **Benefits:** 1. **CORS**: Same parent domain, no cross-origin issues 2. **Clean teardown**: Delete DNS records, R2 bucket, and Pages project together 3. **Professional**: No `.pages.dev` or `.r2.dev` in production URLs ## Implementation Plan ### 1. R2 Bucket Management - [ ] Create R2 bucket on `deploy publish` - [ ] Bucket naming convention: `{project-name}-audio` - [ ] Enable public access via custom domain - [ ] Set appropriate CORS headers ### 2. Custom Domain for R2 - [ ] Create DNS CNAME: `cdn.{domain}` → R2 bucket public URL - [ ] Configure R2 custom domain binding - [ ] Store DNS record IDs for cleanup ### 3. Audio Upload to R2 - [ ] Upload audio files to R2 with S3-compatible API - [ ] Path structure: `{project-name}/audio/track.flac` - [ ] Set appropriate content-type headers (audio/flac, audio/wav, etc.) - [ ] Generate public URLs: `https://cdn.{domain}/{project-name}/audio/track.flac` ### 4. Generator Updates - [ ] Modify `crates/generator` to accept R2 URLs for audio - [ ] Update `player.js` template to use R2 URLs instead of relative paths - [ ] Ensure preview mode still works with local files ### 5. Split Deployment Flow - [ ] Upload audio files → R2 first - [ ] Build static site with R2 URLs embedded - [ ] Deploy HTML/CSS/JS → Pages (small, under limit) - [ ] Configure Pages custom domain ### 6. API-Driven Teardown (No Local State) **Derive everything from album.toml + API queries:** ```rust // Given album.toml + ~/.release-kit/config.toml let project_name = derive_project_name(&album.artist.name, &album.metadata.title); let bucket_name = format!("{}-audio", project_name); let domain = &album.site.domain; // Query Cloudflare API to find resources 1. Query Pages project by name → delete if exists 2. Query R2 bucket by name → delete all objects → delete bucket 3. Query DNS records matching domain pattern → delete all 4. No orphaned resources! ``` **Advantages:** - ✅ No stale state files - ✅ Works across machines - ✅ Self-healing (can always re-derive from album.toml) - ✅ Matches existing `status` command pattern ### 7. Configuration Updates - [ ] Update `deploy configure` to document R2 permissions - [ ] Remove obsolete `r2_bucket` and `pages_project` fields from album.toml - [ ] Keep `subdomain` field for custom domain configuration ### 8. Helper Commands - [ ] Add `deploy list` to show all deployed projects - [ ] Detect orphaned resources (renamed albums) - [ ] Enhanced `deploy status` to show R2 bucket info ## API Token Permissions Update required permissions: ``` Account > Cloudflare Pages > Edit Account > R2 > Edit Zone > DNS > Edit Zone > Zone > Read ``` ## Testing Checklist - [ ] Deploy album with small audio files (< 1MB each) - [ ] Deploy album with large WAV files (> 30MB each) - [ ] Verify audio streaming works from R2 URLs via custom domain - [ ] Verify preview mode still works locally - [ ] Test teardown removes Pages project, R2 bucket, and DNS records - [ ] Test cross-origin requests work correctly - [ ] Test deploy after renaming album in album.toml ## Files to Modify - `crates/cli/src/commands/deploy.rs` - R2 upload logic, split deployment, DNS management - `crates/deployer/src/cloudflare/mod.rs` - R2 API client, DNS API client - `crates/generator/src/lib.rs` - Accept R2 URLs for audio - `crates/cli/src/commands/build.rs` - Pass audio URLs to generator - `crates/core/src/types.rs` - Update CloudflareConfig (remove obsolete fields) ## Open Questions 1. **Audio subdomain**: Use `cdn.{domain}` or `audio.{domain}` or make it configurable? 2. **Bucket lifecycle**: One bucket per album (current plan) or shared bucket? 3. **Partial failure handling**: Retry logic if DNS creation succeeds but R2 upload fails? 4. **Migration path**: How to handle existing (broken) deployments? ## Related This is a critical blocker - the deploy feature is non-functional without R2 support.