feat: добавить базовые сервисы image-platform

- добавлены backend, admin, gateway и worker skeleton
- добавлены Drizzle schema, database package и initial migration
- добавлены shared packages для RabbitMQ topology и S3 helpers
- обновлены dev-инфраструктура, env example, scripts и dependencies
- обновлена документация под versioned image URLs и read-through flow
This commit is contained in:
2026-05-05 09:59:21 +03:00
parent 37592c8b81
commit bcadb85a83
66 changed files with 8698 additions and 213 deletions

View File

@@ -19,12 +19,39 @@ S3_SECRET_ACCESS_KEY=image-password
S3_FORCE_PATH_STYLE=true
# Future local services
PUBLIC_API_BASE_URL=http://localhost:3001
BACKEND_PORT=3001
ADMIN_PORT=5173
GATEWAY_HOST=0.0.0.0
GATEWAY_PORT=8888
PUBLIC_BACKEND_BASE_URL=http://localhost:3001
PUBLIC_ADMIN_BASE_URL=http://localhost:5173
PUBLIC_IMAGE_BASE_URL=http://localhost:8888
# imgproxy is always external for image-platform.
# Local example: run imgproxy separately on localhost:18080.
# Gateway proxies /api and Swagger routes to this upstream.
GATEWAY_BACKEND_UPSTREAM=http://localhost:3001
# Dev imgproxy is exposed only on localhost.
IMGPROXY_PORT=18080
IMGPROXY_UPSTREAM=http://localhost:18080
IMGPROXY_SIGNING_ENABLED=false
IMGPROXY_KEY=
IMGPROXY_SALT=
IMGPROXY_WORKERS=2
IMGPROXY_MAX_SRC_RESOLUTION=20
IMGPROXY_DOWNLOAD_TIMEOUT=30
IMGPROXY_ALLOWED_SOURCES=
# RabbitMQ dev broker is exposed only on localhost.
RABBITMQ_DEFAULT_USER=image
RABBITMQ_DEFAULT_PASS=image-password
RABBITMQ_DEFAULT_VHOST=image_platform
RABBITMQ_PORT=5672
RABBITMQ_MANAGEMENT_PORT=15672
RABBITMQ_URL=amqp://image:image-password@localhost:5672/image_platform
WORKER_PREFETCH=2
# Queue topology
RABBITMQ_JOBS_EXCHANGE=image-platform.jobs
RABBITMQ_GENERATE_VARIANT_QUEUE=image.generate-variant
RABBITMQ_GENERATE_VARIANT_DLX=image-platform.jobs.dlx
RABBITMQ_GENERATE_VARIANT_DLQ=image.generate-variant.dlq

View File

@@ -4,16 +4,18 @@ Image Platform - отдельная площадка для управления
## Статус
Сейчас создан только базовый monorepo и dev-инфраструктура. Приложения `api`, `admin` и `gateway` пока намеренно не созданы.
Сейчас создан базовый monorepo, dev-инфраструктура, NestJS backend с Swagger, чистый Vite React TS admin, Fastify gateway skeleton, Drizzle database package, shared queue/storage packages и worker skeleton.
## Целевая схема
```text
client
-> CDN optional
-> gateway Caddy/Souin hot cache
-> Fastify gateway + L1 memory cache
-> NestJS backend
-> S3/Object Storage persistent variants
-> generator/worker
-> generator/worker on miss
-> RabbitMQ
-> external imgproxy
-> source/original image
```
@@ -27,20 +29,28 @@ client
- PostgreSQL
- MinIO
- MinIO bucket init
- imgproxy dev instance
- RabbitMQ
Позже нодой будут запускаться:
Нодой запускается:
- NestJS API
- worker
- NestJS backend
- React/Vite admin
- Fastify gateway
- worker
Gateway будет добавлен отдельно позже.
Gateway уже добавлен как JS/Fastify skeleton. Сейчас `/images/*` возвращает `501`, пока не подключены DB/S3/imgproxy.
```bash
cp .env.example .env
pnpm install
pnpm infra:up
pnpm db:migrate
pnpm infra:config
pnpm backend:dev
pnpm admin:dev
pnpm gateway:dev
pnpm worker:dev
```
Порты по умолчанию:
@@ -50,11 +60,21 @@ pnpm infra:config
| PostgreSQL | `localhost:5433` |
| MinIO API | `http://localhost:9000` |
| MinIO Console | `http://localhost:9001` |
| imgproxy | `http://localhost:18080` |
| RabbitMQ | `amqp://localhost:5672` |
| RabbitMQ Management | `http://localhost:15672` |
| Backend API | `http://localhost:3001/api` |
| Swagger | `http://localhost:3001/docs` |
| Admin | `http://localhost:5173` |
| Gateway | `http://localhost:8888` |
Если старый локальный `image-gateway` уже занимает `8888`, остановите его или задайте другой `GATEWAY_PORT` в `.env`.
## Документация
- `docs/architecture.md` - целевая архитектура и ответственность компонентов.
- `docs/development.md` - локальный dev flow.
- `docs/data-model.md` - черновик PostgreSQL модели.
- `docs/api-contract-draft.md` - черновик будущего JSON API.
- `docs/data-model.md` - текущая Drizzle/PostgreSQL модель.
- `docs/backend-contract-draft.md` - черновик будущего backend-контракта.
- `docs/imgproxy-contract.md` - контракт с external imgproxy.
- `docs/next-image-provider.md` - контракт custom provider для `next/image`.

12
apps/admin/index.html Normal file
View File

@@ -0,0 +1,12 @@
<!doctype html>
<html lang="ru">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Image Platform Admin</title>
</head>
<body>
<div id="root"></div>
<script type="module" src="/src/main.tsx"></script>
</body>
</html>

24
apps/admin/package.json Normal file
View File

@@ -0,0 +1,24 @@
{
"name": "@image-platform/admin",
"version": "0.1.0",
"private": true,
"type": "module",
"scripts": {
"build": "tsc -b && vite build",
"dev": "vite --host 0.0.0.0 --port 5173",
"preview": "vite preview --host 0.0.0.0 --port 5173",
"typecheck": "tsc -b"
},
"dependencies": {
"react": "^19.2.5",
"react-dom": "^19.2.5"
},
"devDependencies": {
"@types/node": "^24.12.2",
"@types/react": "^19.2.14",
"@types/react-dom": "^19.2.3",
"@vitejs/plugin-react": "^6.0.1",
"typescript": "^5.9.3",
"vite": "^8.0.10"
}
}

111
apps/admin/src/App.css Normal file
View File

@@ -0,0 +1,111 @@
:root {
color: #171411;
background: #f7f4ee;
font-family: Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
font-synthesis: none;
text-rendering: optimizeLegibility;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
}
* {
box-sizing: border-box;
}
body {
min-width: 320px;
min-height: 100vh;
margin: 0;
background:
radial-gradient(circle at 18% 18%, rgba(123, 76, 255, 0.18), transparent 30rem),
#f7f4ee;
}
button,
input,
textarea,
select {
font: inherit;
}
.app-shell {
min-height: 100vh;
padding: 48px 24px;
}
.hero {
max-width: 960px;
margin: 0 auto;
padding: 40px;
border: 1px solid #e4ded4;
border-radius: 32px;
background: rgba(255, 255, 255, 0.82);
box-shadow: 0 22px 80px rgba(40, 32, 21, 0.08);
}
.eyebrow {
margin: 0 0 16px;
color: #7b4cff;
font-size: 13px;
font-weight: 700;
letter-spacing: 0.22em;
text-transform: uppercase;
}
h1 {
max-width: 760px;
margin: 0;
font-size: clamp(40px, 7vw, 76px);
line-height: 0.92;
letter-spacing: -0.06em;
}
.lead {
max-width: 680px;
margin: 24px 0 0;
color: #73695d;
font-size: 18px;
line-height: 1.7;
}
.cards {
display: grid;
grid-template-columns: repeat(3, minmax(0, 1fr));
gap: 16px;
max-width: 960px;
margin: 24px auto 0;
}
.card {
min-height: 150px;
padding: 24px;
border: 1px solid #e4ded4;
border-radius: 24px;
background: rgba(255, 255, 255, 0.76);
}
.card h2 {
margin: 0;
font-size: 18px;
}
.card p {
margin: 12px 0 0;
color: #73695d;
line-height: 1.55;
}
@media (max-width: 720px) {
.app-shell {
padding: 24px 16px;
}
.hero {
padding: 28px;
border-radius: 24px;
}
.cards {
grid-template-columns: 1fr;
}
}

40
apps/admin/src/App.tsx Normal file
View File

@@ -0,0 +1,40 @@
import "./App.css"
const cards = [
{
title: "Assets",
description: "Каталог исходных изображений и связанной metadata.",
},
{
title: "Variants",
description: "Будущие AVIF/WebP/JPEG variants, presets и статусы генерации.",
},
{
title: "Storage",
description: "PostgreSQL как source of truth, S3/MinIO как хранилище bytes.",
},
]
export function App() {
return (
<main className="app-shell">
<section className="hero">
<p className="eyebrow">Image Platform Admin</p>
<h1>чистый Vite React TS app</h1>
<p className="lead">
Это стартовая админка без UI-фреймворков. Дальше сюда добавим управление allowed hosts,
assets, variants и presets.
</p>
</section>
<section className="cards" aria-label="Будущие разделы">
{cards.map((card) => (
<article className="card" key={card.title}>
<h2>{card.title}</h2>
<p>{card.description}</p>
</article>
))}
</section>
</main>
)
}

10
apps/admin/src/main.tsx Normal file
View File

@@ -0,0 +1,10 @@
import { StrictMode } from "react"
import { createRoot } from "react-dom/client"
import { App } from "./App"
createRoot(document.getElementById("root")!).render(
<StrictMode>
<App />
</StrictMode>,
)

View File

@@ -0,0 +1,22 @@
{
"compilerOptions": {
"composite": true,
"tsBuildInfoFile": "./node_modules/.tmp/tsconfig.app.tsbuildinfo",
"target": "ES2022",
"useDefineForClassFields": true,
"lib": ["ES2022", "DOM", "DOM.Iterable"],
"allowJs": false,
"skipLibCheck": true,
"esModuleInterop": true,
"allowSyntheticDefaultImports": true,
"strict": true,
"forceConsistentCasingInFileNames": true,
"module": "ESNext",
"moduleResolution": "Bundler",
"resolveJsonModule": true,
"isolatedModules": true,
"noEmit": true,
"jsx": "react-jsx"
},
"include": ["src"]
}

11
apps/admin/tsconfig.json Normal file
View File

@@ -0,0 +1,11 @@
{
"files": [],
"references": [
{
"path": "./tsconfig.app.json"
},
{
"path": "./tsconfig.node.json"
}
]
}

View File

@@ -0,0 +1,18 @@
{
"compilerOptions": {
"composite": true,
"tsBuildInfoFile": "./node_modules/.tmp/tsconfig.node.tsbuildinfo",
"target": "ES2023",
"lib": ["ES2023"],
"allowJs": false,
"skipLibCheck": true,
"strict": true,
"module": "ESNext",
"moduleResolution": "Bundler",
"allowSyntheticDefaultImports": true,
"isolatedModules": true,
"noEmit": true,
"types": ["node"]
},
"include": ["vite.config.ts"]
}

View File

@@ -0,0 +1,6 @@
import react from "@vitejs/plugin-react"
import { defineConfig } from "vite"
export default defineConfig({
plugins: [react()],
})

View File

@@ -0,0 +1,8 @@
{
"$schema": "https://json.schemastore.org/nest-cli",
"collection": "@nestjs/schematics",
"sourceRoot": "src",
"compilerOptions": {
"deleteOutDir": true
}
}

27
apps/backend/package.json Normal file
View File

@@ -0,0 +1,27 @@
{
"name": "@image-platform/backend",
"version": "0.1.0",
"private": true,
"scripts": {
"build": "nest build",
"dev": "nest start --watch",
"start": "node dist/main.js",
"typecheck": "tsc --noEmit -p tsconfig.json"
},
"dependencies": {
"@nestjs/common": "^11.0.0",
"@nestjs/core": "^11.0.0",
"@nestjs/platform-express": "^11.0.0",
"@nestjs/swagger": "^11.0.0",
"reflect-metadata": "^0.2.2",
"rxjs": "^7.8.1",
"swagger-ui-express": "^5.0.1"
},
"devDependencies": {
"@nestjs/cli": "^11.0.0",
"@nestjs/schematics": "^11.0.0",
"@types/node": "^24.0.0",
"@types/swagger-ui-express": "^4.1.8",
"typescript": "^5.9.0"
}
}

View File

@@ -0,0 +1,9 @@
import { Module } from "@nestjs/common"
import { HealthController } from "./health/health.controller"
import { InternalImagesController } from "./internal-images/internal-images.controller"
@Module({
controllers: [HealthController, InternalImagesController],
})
export class AppModule {}

View File

@@ -0,0 +1,9 @@
import { ApiProperty } from "@nestjs/swagger"
export class HealthResponseDto {
@ApiProperty({ example: "image-platform-api" })
service!: string
@ApiProperty({ example: "ok" })
status!: string
}

View File

@@ -0,0 +1,18 @@
import { Controller, Get } from "@nestjs/common"
import { ApiOkResponse, ApiOperation, ApiTags } from "@nestjs/swagger"
import { HealthResponseDto } from "./health-response.dto"
@ApiTags("system")
@Controller("health")
export class HealthController {
@Get()
@ApiOperation({ summary: "Проверить состояние API" })
@ApiOkResponse({ type: HealthResponseDto })
getHealth(): HealthResponseDto {
return {
service: "image-platform-api",
status: "ok",
}
}
}

View File

@@ -0,0 +1,21 @@
import { ApiProperty } from "@nestjs/swagger"
export class EnsureImageVariantRequestDto {
@ApiProperty({ example: "asset_123" })
assetId!: string
@ApiProperty({ example: 4, minimum: 1 })
version!: number
@ApiProperty({ example: "card" })
preset!: string
@ApiProperty({ example: 640, minimum: 1 })
width!: number
@ApiProperty({ example: 80, minimum: 1 })
quality!: number
@ApiProperty({ enum: ["auto", "avif", "webp", "jpg", "png"], example: "auto" })
format!: "auto" | "avif" | "jpg" | "png" | "webp"
}

View File

@@ -0,0 +1,19 @@
import { Body, Controller, NotImplementedException, Post } from "@nestjs/common"
import { ApiOperation, ApiResponse, ApiTags } from "@nestjs/swagger"
import { EnsureImageVariantRequestDto } from "./ensure-image-variant.dto"
@ApiTags("internal-images")
@Controller("internal/images")
export class InternalImagesController {
@Post("ensure")
@ApiOperation({ summary: "Ensure image variant for Gateway L1 miss" })
@ApiResponse({ status: 501, description: "Read-through image pipeline is not implemented yet" })
ensureImageVariant(@Body() request: EnsureImageVariantRequestDto): never {
throw new NotImplementedException({
message: "image read-through pipeline is not implemented yet",
request,
status: "not_implemented",
})
}
}

37
apps/backend/src/main.ts Normal file
View File

@@ -0,0 +1,37 @@
import { NestFactory } from "@nestjs/core"
import { DocumentBuilder, SwaggerModule } from "@nestjs/swagger"
import { AppModule } from "./app.module"
async function bootstrap() {
const app = await NestFactory.create(AppModule)
app.setGlobalPrefix("api")
app.enableShutdownHooks()
const openApiConfig = new DocumentBuilder()
.setTitle("Image Platform API")
.setDescription("Control plane for image assets, variants, S3 storage and external imgproxy.")
.setVersion("0.1.0")
.addTag("system")
.addTag("assets")
.addTag("variants")
.addTag("allowed-hosts")
.addTag("internal-images")
.build()
const openApiDocument = SwaggerModule.createDocument(app, openApiConfig)
SwaggerModule.setup("docs", app, openApiDocument, {
jsonDocumentUrl: "docs-json",
swaggerOptions: {
persistAuthorization: true,
},
})
const port = Number.parseInt(process.env.API_PORT ?? "3001", 10)
await app.listen(port)
}
void bootstrap()

View File

@@ -0,0 +1,8 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"declaration": true,
"noEmit": false
},
"exclude": ["node_modules", "dist", "test", "**/*.spec.ts"]
}

View File

@@ -0,0 +1,23 @@
{
"compilerOptions": {
"allowSyntheticDefaultImports": true,
"baseUrl": "./",
"declaration": true,
"emitDecoratorMetadata": true,
"esModuleInterop": true,
"experimentalDecorators": true,
"incremental": true,
"module": "commonjs",
"moduleResolution": "node",
"outDir": "./dist",
"removeComments": true,
"skipLibCheck": true,
"sourceMap": true,
"strict": true,
"strictPropertyInitialization": false,
"target": "ES2023",
"types": ["node"]
},
"include": ["src/**/*.ts"],
"exclude": ["node_modules", "dist"]
}

20
apps/gateway/package.json Normal file
View File

@@ -0,0 +1,20 @@
{
"name": "@image-platform/gateway",
"version": "0.1.0",
"private": true,
"type": "module",
"scripts": {
"build": "tsc -p tsconfig.build.json",
"dev": "tsx watch src/main.ts",
"start": "node dist/main.js",
"typecheck": "tsc --noEmit -p tsconfig.json"
},
"dependencies": {
"fastify": "^5.8.5"
},
"devDependencies": {
"@types/node": "^25.6.0",
"tsx": "^4.21.0",
"typescript": "^6.0.3"
}
}

View File

@@ -0,0 +1,27 @@
export type GatewayConfig = {
backendUpstream: URL
host: string
port: number
}
export function loadGatewayConfig(): GatewayConfig {
return {
backendUpstream: new URL(process.env.GATEWAY_BACKEND_UPSTREAM ?? "http://localhost:3001"),
host: process.env.GATEWAY_HOST ?? "0.0.0.0",
port: parsePort(process.env.GATEWAY_PORT, 8888),
}
}
function parsePort(value: string | undefined, fallback: number) {
if (!value) {
return fallback
}
const parsed = Number.parseInt(value, 10)
if (!Number.isFinite(parsed) || parsed < 1 || parsed > 65535) {
throw new Error(`Invalid port: ${value}`)
}
return parsed
}

12
apps/gateway/src/main.ts Normal file
View File

@@ -0,0 +1,12 @@
import { loadGatewayConfig } from "./config.js"
import { createGatewayServer } from "./server.js"
const config = loadGatewayConfig()
const app = createGatewayServer(config)
try {
await app.listen({ host: config.host, port: config.port })
} catch (error) {
app.log.error(error)
process.exit(1)
}

77
apps/gateway/src/proxy.ts Normal file
View File

@@ -0,0 +1,77 @@
import { Readable } from "node:stream"
import type { FastifyReply, FastifyRequest } from "fastify"
const hopByHopHeaders = new Set([
"connection",
"keep-alive",
"proxy-authenticate",
"proxy-authorization",
"te",
"trailer",
"transfer-encoding",
"upgrade",
])
export async function proxyToUpstream(request: FastifyRequest, reply: FastifyReply, upstreamBaseUrl: URL) {
const upstreamUrl = new URL(request.url, upstreamBaseUrl)
const headers = buildProxyHeaders(request)
const init: RequestInit & { duplex?: "half" } = {
headers,
method: request.method,
redirect: "manual",
}
if (request.method !== "GET" && request.method !== "HEAD") {
init.body = request.raw as RequestInit["body"]
init.duplex = "half"
}
const response = await fetch(upstreamUrl, init)
reply.code(response.status)
response.headers.forEach((value, key) => {
if (!hopByHopHeaders.has(key.toLowerCase())) {
reply.header(key, value)
}
})
if (!response.body) {
return reply.send()
}
return reply.send(Readable.fromWeb(response.body))
}
function buildProxyHeaders(request: FastifyRequest) {
const headers = new Headers()
for (const [key, rawValue] of Object.entries(request.headers)) {
const lowerKey = key.toLowerCase()
if (lowerKey === "host" || hopByHopHeaders.has(lowerKey)) {
continue
}
if (Array.isArray(rawValue)) {
for (const value of rawValue) {
headers.append(key, value)
}
continue
}
if (typeof rawValue === "string") {
headers.set(key, rawValue)
}
}
headers.set("x-forwarded-host", request.headers.host ?? "")
headers.set("x-forwarded-proto", "http")
if (request.ip) {
headers.set("x-forwarded-for", request.ip)
}
return headers
}

View File

@@ -0,0 +1,90 @@
import Fastify from "fastify"
import type { GatewayConfig } from "./config.js"
import { proxyToUpstream } from "./proxy.js"
export function createGatewayServer(config: GatewayConfig) {
const app = Fastify({
logger: true,
})
app.get("/health", async () => ({
service: "image-platform-gateway",
status: "ok",
}))
app.get<{ Params: ImageParams; Querystring: ImageQuery }>(
"/images/:assetId/:version/:preset",
async (request, reply) => {
const version = parseVersionParam(request.params.version)
if (version === null) {
return reply.code(400).send({
message: "image version must use v{number} format",
statusCode: 400,
})
}
const width = parseOptionalInteger(request.query.w)
const quality = parseOptionalInteger(request.query.q)
const format = request.query.f ?? "auto"
return reply.code(501).header("cache-control", "no-store").send({
assetId: request.params.assetId,
format,
message: "image gateway read-through pipeline is not implemented yet",
preset: request.params.preset,
quality,
status: "not_implemented",
version,
width,
})
},
)
app.all("/api/*", async (request, reply) => proxyToUpstream(request, reply, config.backendUpstream))
app.all("/docs", async (request, reply) => proxyToUpstream(request, reply, config.backendUpstream))
app.all("/docs/*", async (request, reply) => proxyToUpstream(request, reply, config.backendUpstream))
app.all("/docs-json", async (request, reply) => proxyToUpstream(request, reply, config.backendUpstream))
app.setNotFoundHandler(async (_request, reply) => {
return reply.code(404).send({
message: "route not found",
statusCode: 404,
})
})
return app
}
type ImageQuery = {
f?: string
q?: string
w?: string
}
type ImageParams = {
assetId: string
preset: string
version: string
}
function parseVersionParam(value: string) {
if (!value.startsWith("v")) {
return null
}
const parsed = Number.parseInt(value.slice(1), 10)
return Number.isSafeInteger(parsed) && parsed > 0 ? parsed : null
}
function parseOptionalInteger(value: string | undefined) {
if (!value) {
return null
}
const parsed = Number.parseInt(value, 10)
return Number.isFinite(parsed) ? parsed : null
}

View File

@@ -0,0 +1,6 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"noEmit": false
}
}

View File

@@ -0,0 +1,22 @@
{
"compilerOptions": {
"allowSyntheticDefaultImports": true,
"declaration": true,
"esModuleInterop": true,
"forceConsistentCasingInFileNames": true,
"lib": ["ES2023"],
"module": "NodeNext",
"moduleResolution": "NodeNext",
"noUncheckedIndexedAccess": true,
"outDir": "./dist",
"rootDir": "./src",
"resolveJsonModule": true,
"skipLibCheck": true,
"sourceMap": true,
"strict": true,
"target": "ES2023",
"types": ["node"]
},
"include": ["src/**/*.ts"],
"exclude": ["dist", "node_modules"]
}

24
apps/worker/package.json Normal file
View File

@@ -0,0 +1,24 @@
{
"name": "@image-platform/worker",
"version": "0.1.0",
"private": true,
"type": "module",
"scripts": {
"build": "tsc -p tsconfig.build.json",
"dev": "tsx watch src/main.ts",
"start": "node dist/main.js",
"typecheck": "tsc --noEmit -p tsconfig.json"
},
"dependencies": {
"@image-platform/database": "workspace:*",
"@image-platform/queue": "workspace:*",
"@image-platform/storage": "workspace:*",
"amqplib": "^1.0.4"
},
"devDependencies": {
"@types/amqplib": "^0.10.8",
"@types/node": "^25.6.0",
"tsx": "^4.21.0",
"typescript": "^6.0.3"
}
}

25
apps/worker/src/config.ts Normal file
View File

@@ -0,0 +1,25 @@
import { loadQueueTopologyFromEnv, type QueueTopology } from "@image-platform/queue"
export type WorkerConfig = {
prefetch: number
queueTopology: QueueTopology
rabbitmqUrl: string
}
export function loadWorkerConfig(env: NodeJS.ProcessEnv = process.env): WorkerConfig {
return {
prefetch: parsePositiveInteger(env.WORKER_PREFETCH, 2),
queueTopology: loadQueueTopologyFromEnv(env),
rabbitmqUrl: env.RABBITMQ_URL ?? "amqp://image:image-password@localhost:5672/image_platform",
}
}
function parsePositiveInteger(value: string | undefined, fallback: number) {
if (!value) {
return fallback
}
const parsed = Number.parseInt(value, 10)
return Number.isSafeInteger(parsed) && parsed > 0 ? parsed : fallback
}

67
apps/worker/src/main.ts Normal file
View File

@@ -0,0 +1,67 @@
import amqp, { type Channel, type ConsumeMessage } from "amqplib"
import { parseGenerateVariantJobBuffer, type QueueTopology } from "@image-platform/queue"
import { loadWorkerConfig } from "./config.js"
async function bootstrap() {
const config = loadWorkerConfig()
const connection = await amqp.connect(config.rabbitmqUrl)
const channel = await connection.createChannel()
await assertQueueTopology(channel, config.queueTopology)
await channel.prefetch(config.prefetch)
await channel.consume(
config.queueTopology.generateVariantQueue,
(message) => void handleGenerateVariantMessage(channel, message),
{ noAck: false },
)
console.log(`worker consuming ${config.queueTopology.generateVariantQueue}`)
const shutdown = async () => {
console.log("worker shutting down")
await channel.close().catch((error: unknown) => console.error("failed to close RabbitMQ channel", error))
await connection.close().catch((error: unknown) => console.error("failed to close RabbitMQ connection", error))
process.exit(0)
}
process.once("SIGINT", () => void shutdown())
process.once("SIGTERM", () => void shutdown())
}
async function assertQueueTopology(channel: Channel, topology: QueueTopology) {
await channel.assertExchange(topology.jobsExchange, "direct", { durable: true })
await channel.assertExchange(topology.jobsDeadLetterExchange, "direct", { durable: true })
await channel.assertQueue(topology.generateVariantQueue, {
deadLetterExchange: topology.jobsDeadLetterExchange,
deadLetterRoutingKey: topology.generateVariantDeadLetterRoutingKey,
durable: true,
})
await channel.assertQueue(topology.generateVariantDeadLetterQueue, { durable: true })
await channel.bindQueue(topology.generateVariantQueue, topology.jobsExchange, topology.generateVariantRoutingKey)
await channel.bindQueue(
topology.generateVariantDeadLetterQueue,
topology.jobsDeadLetterExchange,
topology.generateVariantDeadLetterRoutingKey,
)
}
async function handleGenerateVariantMessage(channel: Channel, message: ConsumeMessage | null) {
if (message === null) {
return
}
try {
const job = parseGenerateVariantJobBuffer(message.content)
console.log("generate variant job received, handler not implemented yet", job)
channel.nack(message, false, false)
} catch (error) {
console.error("invalid generate variant job", error)
channel.nack(message, false, false)
}
}
void bootstrap().catch((error: unknown) => {
console.error("worker failed to start", error)
process.exit(1)
})

View File

@@ -0,0 +1,7 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"noEmit": false
},
"exclude": ["dist", "node_modules", "**/*.spec.ts"]
}

21
apps/worker/tsconfig.json Normal file
View File

@@ -0,0 +1,21 @@
{
"compilerOptions": {
"allowSyntheticDefaultImports": true,
"declaration": true,
"esModuleInterop": true,
"forceConsistentCasingInFileNames": true,
"lib": ["ES2023"],
"module": "NodeNext",
"moduleResolution": "NodeNext",
"noUncheckedIndexedAccess": true,
"outDir": "./dist",
"rootDir": "./src",
"skipLibCheck": true,
"sourceMap": true,
"strict": true,
"target": "ES2023",
"types": ["node"]
},
"include": ["src/**/*.ts"],
"exclude": ["dist", "node_modules"]
}

View File

@@ -1,124 +0,0 @@
# Черновик API Контракта
Это не реализация API, а фиксация будущего контракта для NestJS backend.
Backend отдаёт JSON, metadata, statuses и URLs. Он не должен проксировать image bytes на каждый обычный запрос.
## Allowed Hosts
```text
GET /allowed-hosts
POST /allowed-hosts
PATCH /allowed-hosts/:id
DELETE /allowed-hosts/:id
```
## Assets
```text
GET /assets
POST /assets
GET /assets/:id
DELETE /assets/:id
```
`POST /assets` request:
```json
{
"sourceUrl": "https://example.com/photo.jpg"
}
```
Responsibilities:
- validate source URL;
- check `allowed_image_hosts`;
- create or reuse `image_assets` row;
- optionally save original to S3 later.
## Variants
```text
GET /assets/:id/variants
POST /assets/:id/variants
POST /variants/:id/regenerate
DELETE /variants/:id
```
`POST /assets/:id/variants` request:
```json
{
"preset": "card",
"format": "webp",
"width": 640
}
```
Response if ready:
```json
{
"id": "variant_123",
"status": "ready",
"url": "http://localhost:8888/images/asset_123/w640_q80_card.webp"
}
```
Response if generation is async:
```json
{
"id": "variant_123",
"status": "pending",
"url": null
}
```
## Image URLs For UI
Для UI нужен endpoint, который возвращает готовый набор URLs для `<picture>`/`srcset`.
```text
GET /assets/:id/picture?preset=card
```
Example response:
```json
{
"assetId": "asset_123",
"preset": "card",
"sources": [
{
"type": "image/avif",
"srcset": "http://localhost:8888/images/asset_123/w320_card.avif 320w, http://localhost:8888/images/asset_123/w640_card.avif 640w"
},
{
"type": "image/webp",
"srcset": "http://localhost:8888/images/asset_123/w320_card.webp 320w, http://localhost:8888/images/asset_123/w640_card.webp 640w"
}
],
"fallback": {
"src": "http://localhost:8888/images/asset_123/w640_card.jpg",
"width": 640,
"height": null
}
}
```
## Worker Lifecycle
Первый MVP может генерировать sync на request. Если генерация тяжёлая, variant создаётся как `pending`, а worker обрабатывает job.
PostgreSQL может выступить первой очередью:
```text
SELECT * FROM image_variants
WHERE status = 'pending'
FOR UPDATE SKIP LOCKED
LIMIT 1
```
Позже можно добавить Redis/Valkey или отдельную queue, если PostgreSQL станет узким местом.

View File

@@ -12,24 +12,49 @@
|---|---|---|
| PostgreSQL | сейчас | Источник правды для assets, variants, hosts, statuses |
| S3/MinIO | сейчас | Хранилище originals и generated variants |
| API | позже | JSON API, admin operations, validation, orchestration |
| Worker | позже | Генерация variants, upload в S3, update PostgreSQL |
| Admin UI | позже | Управление hosts/assets/variants/presets |
| Gateway | позже | Caddy/Souin hot cache и delivery layer |
| Backend | сейчас | NestJS JSON API, Swagger, PostgreSQL/S3/RabbitMQ orchestration |
| Worker | позже | RabbitMQ consumer, imgproxy processing, upload в S3, update PostgreSQL |
| Admin UI | сейчас | React/Vite UI для будущего управления hosts/assets/variants/presets |
| Gateway | сейчас | Fastify public image origin, L1 memory cache, root routing, без DB/S3 доступа |
| RabbitMQ | сейчас | Очередь задач генерации variants |
| imgproxy | external | CPU-heavy image processing |
## Архитектурное решение
Нужное поведение - Cloudinary-like: публичный URL изображения сам запускает read-through pipeline, если variant ещё не создан.
`image-platform` строится ради совместимости с `next/image` как custom loader provider. Next.js application не должен заранее вызывать API, ждать генерацию и затем подставлять S3 URL. Он должен передавать `src`, `width` и `quality` в loader, а loader должен вернуть стабильный URL нашего image origin.
Gateway поэтому является обязательной частью public delivery path, а не опциональным кешем.
## Целевая delivery схема
```text
client
-> CDN optional
-> gateway Caddy/Souin
-> S3 ready variant
-> generator fallback
-> Fastify gateway L1 memory cache
-> NestJS backend
-> PostgreSQL + S3 ready variant
-> RabbitMQ -> worker
-> external imgproxy
-> source image
```
Read-through flow:
```text
1. client запрашивает /images/{assetId}/v{version}/{preset}?w=640&q=80&f=auto
2. CDN HIT -> ответ сразу
3. Gateway L1 HIT -> ответ сразу
4. Gateway L1 MISS -> Gateway вызывает Backend internal ensure endpoint
5. Backend проверяет PostgreSQL/S3
6. S3 HIT -> Backend стримит bytes Gateway, Gateway кладёт ответ в L1
7. S3 MISS -> Backend ставит RabbitMQ job
8. Worker вызывает external imgproxy и сохраняет результат в S3
9. Worker обновляет PostgreSQL, Backend отдаёт готовые bytes Gateway
10. Gateway кладёт ответ в L1 и отдаёт клиенту
```
## Разделение ответственности
PostgreSQL отвечает на вопросы:
@@ -49,29 +74,45 @@ S3 хранит байты:
Gateway отдаёт картинки:
- hot cache HIT - сразу из Souin;
- cache MISS - из S3;
- S3 MISS - через generator fallback.
- L1 memory HIT - сразу из памяти;
- L1 memory MISS - вызывает Backend;
- не имеет доступа к PostgreSQL, S3 и RabbitMQ.
Backend не должен проксировать картинки на каждый обычный запрос. Он отдаёт JSON, статусы и URLs.
Backend управляет процессами: PostgreSQL, S3 read, RabbitMQ enqueue, ожидание ready variant. Worker выполняет CPU-heavy generation через external imgproxy и пишет результат в S3.
## URL модель
Публичные URL должны быть стабильными и не раскрывать source URL:
Публичные URL должны быть стабильными и не раскрывать source URL. Для Next/image provider основной URL должен принимать width/quality из loader:
```text
/images/{assetId}/{variantHash}.{format}
/images/{assetId}/v{version}/{preset}?w={width}&q={quality}&f=auto
```
Примеры:
```text
/images/asset_123/w640_q80_cfill.avif
/images/asset_123/w640_q80_cfill.webp
/images/asset_123/w640_q80_cfill.jpg
/images/asset_123/v4/card?w=640&q=80&f=auto
/images/asset_123/v4/hero?w=1920&q=80&f=auto
```
Формат лучше делать явным в URL и отдавать через `<picture>`/`srcset`, а не выбирать по `Accept` header. Так CDN/S3/Gateway cache остаётся предсказуемым.
`v{version}` берётся из `image_assets.current_version` и меняется при обновлении source image. Это даёт immutable cache без purge старых CDN/L1/S3 keys.
`f=auto` нужен для совместимости с `next/image` custom loader: Next передаёт в loader `src`, `width` и `quality`, но не выбирает AVIF/WebP сам при custom loader. Image origin должен выбрать формат по `Accept` header, как Cloudinary `f_auto`.
Из-за `f=auto` обязательно:
- S3 key должен включать фактически выбранный формат;
- response должен выставлять `Vary: Accept`;
- CDN и Gateway L1 cache должны учитывать `Accept`;
- response должен выставлять `Cache-Control: public, max-age=31536000, immutable` для versioned assets.
Для ручного `<picture>`/`srcset` можно добавить явный формат позже:
```text
/images/{assetId}/v{version}/{preset}?w=640&q=80&f=avif
/images/{assetId}/v{version}/{preset}?w=640&q=80&f=webp
/images/{assetId}/v{version}/{preset}?w=640&q=80&f=jpg
```
## External imgproxy

View File

@@ -0,0 +1,224 @@
# Черновик Backend Контракта
Это черновик будущего бизнес-контракта для NestJS backend. Сейчас реализованы system endpoints, Swagger и placeholder для internal image ensure.
## System
```text
GET /api/health
GET /docs
GET /docs-json
```
NestJS backend отдаёт JSON, metadata, statuses и URLs. Public image origin находится в Fastify Gateway. Backend владеет PostgreSQL, S3 orchestration и RabbitMQ jobs.
## Internal Image Ensure
Этот internal endpoint вызывается Gateway на L1 miss. Gateway не ходит в DB/S3 напрямую.
```text
POST /api/internal/images/ensure
```
Текущий статус реализации - endpoint зарезервирован и возвращает `501 Not Implemented`. Gateway `/images/*` пока тоже возвращает placeholder `501 Not Implemented`.
Request body:
```json
{
"assetId": "asset_123",
"version": 4,
"preset": "card",
"width": 640,
"quality": 80,
"format": "webp"
}
```
Query params:
| Param | Описание |
|---|---|
| `w` | целевая ширина, должна быть разрешена preset или округлена до ближайшей разрешённой |
| `q` | качество, должно быть из allowlist качества |
| `f` | `auto`, `avif`, `webp`, `jpg`; для Next/image по умолчанию `auto` |
Responsibilities:
- проверить `assetId` и `preset`;
- вычислить deterministic `variantHash`;
- проверить PostgreSQL и S3;
- если variant готов в S3, вернуть bytes или stream metadata для Gateway;
- если variant отсутствует, создать/переиспользовать variant row;
- поставить `generate-variant` job в RabbitMQ;
- дождаться `ready` до timeout, чтобы первый `next/image` request мог получить картинку;
- вернуть image response или metadata для Gateway.
Response headers:
```http
Content-Type: image/avif | image/webp | image/jpeg
Cache-Control: public, max-age=31536000, immutable
Vary: Accept
ETag: "..."
```
Ошибки:
| Status | Когда |
|---|---|
| `400` | некорректные query params |
| `404` | asset или preset не найден |
| `409` | variant уже генерируется и sync ожидание отключено |
| `422` | source image нельзя обработать |
| `502` | external imgproxy недоступен |
Public endpoint реализуется в Fastify Gateway, internal ensure endpoint - в Backend:
```text
client -> CDN -> Gateway /images/* -> Backend ensure -> RabbitMQ -> Worker -> imgproxy -> S3
```
Важно: `/images/*` не должен жить в NestJS `/api`. Gateway остаётся public image origin, Backend остаётся управлением процессов.
Public image URL versioned: `/images/{assetId}/v{version}/{preset}?w={width}&q={quality}&f=auto`.
## Allowed Hosts
```text
GET /allowed-hosts
POST /allowed-hosts
PATCH /allowed-hosts/:id
DELETE /allowed-hosts/:id
```
## Assets
```text
GET /assets
POST /assets
GET /assets/:id
DELETE /assets/:id
```
`POST /assets` request:
```json
{
"sourceUrl": "https://example.com/photo.jpg"
}
```
Responsibilities:
- validate source URL;
- check `allowed_image_hosts`;
- create or reuse `image_assets` row;
- optionally save original to S3 later.
## Variants
```text
GET /assets/:id/variants
POST /assets/:id/variants
POST /variants/:id/regenerate
DELETE /variants/:id
```
`POST /assets/:id/variants` request:
```json
{
"preset": "card",
"format": "webp",
"width": 640
}
```
Response if ready:
```json
{
"id": "variant_123",
"status": "ready",
"url": "http://localhost:8888/images/asset_123/v4/card?w=640&q=80&f=webp"
}
```
Response if generation is async:
```json
{
"id": "variant_123",
"status": "pending",
"url": null
}
```
## Image URLs For UI
Для ручного UI можно добавить endpoint, который возвращает готовый набор URLs для `<picture>`/`srcset`. Для `next/image` основным контрактом остаётся custom loader из `docs/next-image-provider.md`.
```text
GET /assets/:id/picture?preset=card
```
Example response:
```json
{
"assetId": "asset_123",
"preset": "card",
"sources": [
{
"type": "image/avif",
"srcset": "http://localhost:8888/images/asset_123/v4/card?w=320&q=80&f=avif 320w, http://localhost:8888/images/asset_123/v4/card?w=640&q=80&f=avif 640w"
},
{
"type": "image/webp",
"srcset": "http://localhost:8888/images/asset_123/v4/card?w=320&q=80&f=webp 320w, http://localhost:8888/images/asset_123/v4/card?w=640&q=80&f=webp 640w"
}
],
"fallback": {
"src": "http://localhost:8888/images/asset_123/v4/card?w=640&q=80&f=jpg",
"width": 640,
"height": null
}
}
```
## Worker Lifecycle
Worker выполняет задачи из RabbitMQ. Задачи создаёт Backend.
RabbitMQ topology:
```text
exchange: image-platform.jobs
queue: image.generate-variant
dead-letter exchange: image-platform.jobs.dlx
dead-letter queue: image.generate-variant.dlq
```
Job payload должен быть минимальным:
```json
{
"jobId": "job_123",
"variantId": "variant_123"
}
```
Worker читает детали variant из PostgreSQL, вызывает imgproxy, пишет результат в S3 и обновляет status в PostgreSQL.
Если генерация тяжёлая или не успела завершиться до timeout, Backend может вернуть Gateway `504`, а job продолжит выполняться/retry по очереди.
PostgreSQL может выступить первой очередью:
```text
SELECT * FROM image_variants
WHERE status = 'pending'
FOR UPDATE SKIP LOCKED
LIMIT 1
```
PostgreSQL-backed очередь не используется как основной механизм: для jobs выбран RabbitMQ.

View File

@@ -1,16 +1,16 @@
# Черновик Data Model
# Data Model
Это черновик для будущих миграций PostgreSQL. Реальные таблицы добавим вместе с API.
Текущая PostgreSQL модель описана в `packages/database/src/schema.ts` и миграциях Drizzle в `packages/database/drizzle`.
## allowed_image_hosts
```text
id
hostname
enabled
description nullable
created_at
updated_at
id uuid pk default gen_random_uuid()
hostname text not null unique
enabled boolean not null default true
description text nullable
created_at timestamptz not null default now()
updated_at timestamptz not null default now()
```
Правила normalization:
@@ -26,53 +26,119 @@ updated_at
## image_assets
```text
id
source_url
source_host
source_hash
original_s3_key nullable
status
width nullable
height nullable
content_type nullable
size_bytes nullable
created_at
updated_at
id uuid pk default gen_random_uuid()
public_id text not null unique
current_version integer not null default 1
status asset_status not null default active
created_at timestamptz not null default now()
updated_at timestamptz not null default now()
```
`public_id` - стабильный идентификатор в public URL. `current_version` указывает активную версию source image и используется для cache invalidation без purge.
## image_asset_versions
```text
id uuid pk default gen_random_uuid()
asset_id uuid not null references image_assets(id) on delete cascade
version integer not null
source_url text not null
source_host text not null
source_hash text not null
original_s3_key text nullable
width integer nullable
height integer nullable
content_type text nullable
size_bytes bigint nullable
created_at timestamptz not null default now()
```
Каждое изменение source image создаёт новую версию. Старые versioned URLs остаются immutable, новые клиенты получают URL с новым `v{version}`.
## image_variants
```text
id
asset_id
preset
variant_hash
format
width
height nullable
quality
s3_key
status: pending | processing | ready | failed
size_bytes nullable
error nullable
created_at
updated_at
last_accessed_at nullable
id uuid pk default gen_random_uuid()
asset_id uuid not null references image_assets(id) on delete cascade
asset_version_id uuid not null references image_asset_versions(id) on delete cascade
asset_version integer not null
preset text not null
variant_hash text not null unique
requested_format requested_format not null default auto
format variant_format not null
width integer not null
height integer nullable
quality integer not null
s3_key text not null unique
content_type text nullable
etag text nullable
status variant_status not null default pending
size_bytes bigint nullable
error text nullable
attempt_count integer not null default 0
last_accessed_at timestamptz nullable
created_at timestamptz not null default now()
updated_at timestamptz not null default now()
```
`requested_format` хранит то, что запросил клиент (`auto`, `avif`, `webp`, `jpg`, `png`). `format` хранит фактический output format после negotiation по `Accept`.
## Enums
```text
asset_status: active | disabled | deleted
requested_format: auto | avif | webp | jpg | png
variant_format: avif | webp | jpg | png
variant_status: pending | processing | ready | failed
```
## Unique constraints
```text
allowed_image_hosts(hostname)
image_assets(source_hash)
image_variants(asset_id, variant_hash, format)
image_assets(public_id)
image_asset_versions(asset_id, version)
image_variants(asset_id, asset_version, preset, width, quality, format)
image_variants(s3_key)
image_variants(variant_hash)
```
Индексы:
```text
image_asset_versions(source_hash)
image_variants(status)
```
## S3 layout
```text
originals/{assetId}/source
variants/{assetId}/{variantHash}.{format}
originals/{assetId}/v{version}/source
variants/{assetId}/v{version}/{variantHash}.{format}
```
`variantHash` должен включать:
- `assetId`;
- `assetVersion`;
- `preset`;
- normalized width;
- normalized quality;
- фактический output format;
- параметры transform, влияющие на bytes.
Для `f=auto` в public URL в S3 всё равно пишется фактический формат:
```text
variants/asset_123/v4/card_w640_q80_avif.avif
variants/asset_123/v4/card_w640_q80_webp.webp
variants/asset_123/v4/card_w640_q80_jpg.jpg
```
Public URL также versioned:
```text
/images/{assetId}/v{version}/{preset}?w={width}&q={quality}&f=auto
```
## Presets

View File

@@ -2,15 +2,18 @@
## Принцип
В Docker запускаем стабильную инфраструктуру. Кодовые сервисы позже запускаем нодой с hot reload.
В Docker запускаем стабильную инфраструктуру. Кодовые сервисы запускаем нодой с hot reload.
Сейчас в Docker есть только:
- PostgreSQL;
- MinIO;
- MinIO bucket init.
- MinIO bucket init;
- imgproxy dev instance;
- RabbitMQ.
`api`, `worker`, `admin` и `gateway` пока не созданы.
`backend` уже создан как NestJS app. `admin` уже создан как чистый Vite React TS app. `gateway` уже создан как Fastify app. `worker` уже создан как RabbitMQ skeleton.
Gateway обязателен для Cloudinary-like поведения и интеграции с `next/image`.
## Запуск инфраструктуры
@@ -45,40 +48,139 @@ pnpm infra:logs
| PostgreSQL | `localhost:5433` |
| MinIO API | `http://localhost:9000` |
| MinIO Console | `http://localhost:9001` |
| imgproxy | `http://localhost:18080` |
| RabbitMQ | `amqp://localhost:5672` |
| RabbitMQ Management | `http://localhost:15672` |
| Backend API | `http://localhost:3001/api` |
| Swagger | `http://localhost:3001/docs` |
| Admin | `http://localhost:5173` |
| Gateway | `http://localhost:8888` |
Если `localhost:8888` занят старым `image-gateway`, остановите старый stack или задайте `GATEWAY_PORT=8890` в `.env`.
## Backend
Запустить NestJS backend:
```bash
pnpm backend:dev
```
Проверки:
```bash
curl http://localhost:3001/api/health
open http://localhost:3001/docs
```
## Database
Drizzle schema живёт в `packages/database/src/schema.ts`, миграции - в `packages/database/drizzle`.
Сгенерировать миграцию после изменения schema:
```bash
pnpm db:generate
```
Применить миграции к локальному PostgreSQL:
```bash
pnpm db:migrate
```
Открыть Drizzle Studio из корня проекта:
```bash
pnpm db:studio
```
Проверить database package:
```bash
pnpm db:typecheck
pnpm db:build
```
## Admin
Запустить React/Vite admin:
```bash
pnpm admin:dev
```
Открыть:
```bash
open http://localhost:5173
```
## Gateway
Gateway запускается нодой:
```bash
pnpm gateway:dev
```
Проверить gateway health:
```bash
curl http://localhost:8888/health
```
Проверить placeholder image origin route:
```bash
curl -i "http://localhost:8888/images/asset_123/v4/card?w=640&q=80&f=auto"
```
Ожидаемый текущий статус - `501`, потому что S3/imgproxy генерация ещё не реализована.
## Worker
Worker запускается нодой и сейчас только объявляет RabbitMQ topology, слушает `image.generate-variant` и отправляет полученные jobs в DLQ, потому что генерация ещё не реализована.
```bash
pnpm worker:dev
```
Проверить worker package без запуска consumer:
```bash
pnpm worker:typecheck
pnpm worker:build
```
## Будущий dev flow
Когда появятся приложения:
Текущая и будущая схема:
```text
React/Vite admin localhost:5173
-> NestJS API localhost:3001
-> NestJS backend localhost:3001
-> PostgreSQL localhost:5433
-> MinIO localhost:9000
-> RabbitMQ localhost:5672
worker node process
-> PostgreSQL
-> MinIO
-> external imgproxy
-> RabbitMQ
-> imgproxy localhost:18080
gateway Caddy/Souin localhost:8888
-> S3/MinIO ready variant
-> API/generator fallback on host.docker.internal:3001
Fastify gateway localhost:8888
-> L1 memory cache
-> Backend internal ensure endpoint
```
Для Linux gateway container должен видеть host services через:
## imgproxy для разработки
```yaml
extra_hosts:
- "host.docker.internal:host-gateway"
```
## External imgproxy для разработки
`imgproxy` не входит в `image-platform` stack. Для локальной разработки можно использовать любой внешний endpoint и прописать его в `.env`:
В dev compose поднимается локальный `imgproxy`, опубликованный только на `127.0.0.1:18080`:
```env
IMGPROXY_UPSTREAM=http://localhost:18080
```
Если нужен локальный standalone imgproxy, его можно запустить отдельно вне этого compose stack. Он остаётся внешней зависимостью платформы.
Для production `imgproxy` всё равно рассматривается как внешняя зависимость и может жить на отдельном мощном сервере.

View File

@@ -63,7 +63,7 @@ Final signed URL:
## Security rules
- Не отдавать `IMGPROXY_KEY` и `IMGPROXY_SALT` в браузер.
- Source URL валидировать в API/worker.
- Source URL валидировать в Backend/worker.
- Разрешать только `http` и `https`.
- Запрещать localhost, private IP, loopback, link-local.
- Source host должен быть enabled в `allowed_image_hosts`.

105
docs/next-image-provider.md Normal file
View File

@@ -0,0 +1,105 @@
# Next/image Provider
`image-platform` должен работать как custom image provider для `next/image`.
## Next.js contract
Next.js custom loader получает только:
- `src`;
- `width`;
- `quality`.
Поэтому public image URL должен принимать эти параметры напрямую и сам запускать read-through генерацию при miss.
## Loader config
В Next.js приложении используется `loaderFile`:
```js
// next.config.js
module.exports = {
images: {
loader: "custom",
loaderFile: "./src/image-platform-loader.js",
qualities: [60, 75, 80, 90],
},
}
```
Пример loader:
```js
"use client"
const imageBaseUrl = process.env.NEXT_PUBLIC_IMAGE_PLATFORM_URL
export default function imagePlatformLoader({ src, width, quality }) {
const normalizedSrc = src.startsWith("/") ? src.slice(1) : src
const q = quality || 80
return `${imageBaseUrl}/images/${normalizedSrc}?w=${width}&q=${q}&f=auto`
}
```
Пример использования:
```tsx
import Image from "next/image"
export function ProductCard() {
return <Image src="asset_123/v4/card" width={640} height={420} alt="Product" />
}
```
## Public URL
```text
GET /images/{assetId}/v{version}/{preset}?w={width}&q={quality}&f=auto
```
Сейчас route уже зарезервирован в Fastify Gateway, но возвращает placeholder `501`, пока не реализованы PostgreSQL/S3/imgproxy read-through операции.
Пример:
```text
https://img.example.com/images/asset_123/v4/card?w=640&q=80&f=auto
```
`src` не должен быть source URL. Это должен быть versioned platform identifier, например `asset_123/v4/card`. Source URL хранится в PostgreSQL и не раскрывается в public image URL.
`v{version}` меняется при обновлении source image. Старые URL можно кэшировать как immutable без purge.
## Format auto
`f=auto` выбирает output format по `Accept` header:
1. `image/avif`, если клиент поддерживает AVIF и preset разрешает AVIF.
2. `image/webp`, если клиент поддерживает WebP и preset разрешает WebP.
3. `image/jpeg` или original fallback.
Для auto format обязательны headers:
```http
Vary: Accept
Cache-Control: public, max-age=31536000, immutable
Content-Type: image/avif | image/webp | image/jpeg
```
CDN и Gateway L1 cache должны учитывать `Vary: Accept`, иначе можно отдать AVIF клиенту без AVIF support.
## Read-through behavior
```text
client -> CDN -> Fastify gateway -> L1 memory -> Backend -> RabbitMQ -> Worker -> imgproxy -> S3
```
Поведение:
- CDN HIT: backend не вызывается.
- Gateway L1 HIT: backend не вызывается.
- Gateway L1 MISS: Gateway вызывает Backend internal ensure endpoint.
- S3 HIT: Backend отдаёт bytes Gateway, Gateway кладёт result в L1.
- S3 MISS: Backend ставит RabbitMQ job, Worker генерирует variant через external imgproxy, сохраняет в S3, обновляет PostgreSQL, Backend возвращает bytes Gateway.
Так достигается Cloudinary-like поведение: первый запрос создаёт derived asset, следующие запросы отдаются из cache/storage.

View File

@@ -46,6 +46,33 @@ services:
mc anonymous set download local/$${S3_BUCKET}"
restart: "no"
imgproxy:
image: darthsim/imgproxy:latest
restart: unless-stopped
ports:
- "127.0.0.1:${IMGPROXY_PORT:-18080}:8080"
environment:
GODEBUG: http2client=0
IMGPROXY_KEY: ${IMGPROXY_KEY:-}
IMGPROXY_SALT: ${IMGPROXY_SALT:-}
IMGPROXY_WORKERS: ${IMGPROXY_WORKERS:-2}
IMGPROXY_MAX_SRC_RESOLUTION: ${IMGPROXY_MAX_SRC_RESOLUTION:-20}
IMGPROXY_USE_ETAG: "true"
IMGPROXY_ENABLE_VIDEO_THUMBNAILS: "false"
IMGPROXY_DOWNLOAD_TIMEOUT: ${IMGPROXY_DOWNLOAD_TIMEOUT:-30}
IMGPROXY_ALLOWED_SOURCES: ${IMGPROXY_ALLOWED_SOURCES:-}
rabbitmq:
image: rabbitmq:4-management-alpine
restart: unless-stopped
environment:
RABBITMQ_DEFAULT_USER: ${RABBITMQ_DEFAULT_USER:-image}
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_DEFAULT_PASS:-image-password}
RABBITMQ_DEFAULT_VHOST: ${RABBITMQ_DEFAULT_VHOST:-image_platform}
ports:
- "127.0.0.1:${RABBITMQ_PORT:-5672}:5672"
- "127.0.0.1:${RABBITMQ_MANAGEMENT_PORT:-15672}:15672"
volumes:
postgres-data:
minio-data:

View File

@@ -9,10 +9,35 @@
"pnpm": ">=10.0.0"
},
"scripts": {
"admin:build": "pnpm --filter @image-platform/admin build",
"admin:dev": "pnpm --filter @image-platform/admin dev",
"admin:preview": "pnpm --filter @image-platform/admin preview",
"admin:typecheck": "pnpm --filter @image-platform/admin typecheck",
"backend:build": "pnpm --filter @image-platform/backend build",
"backend:dev": "pnpm --filter @image-platform/backend dev",
"backend:start": "pnpm --filter @image-platform/backend start",
"backend:typecheck": "pnpm --filter @image-platform/backend typecheck",
"db:build": "pnpm --filter @image-platform/database build",
"db:generate": "pnpm --filter @image-platform/database db:generate",
"db:migrate": "pnpm --filter @image-platform/database db:migrate",
"db:studio": "pnpm --filter @image-platform/database db:studio",
"db:typecheck": "pnpm --filter @image-platform/database typecheck",
"gateway:build": "pnpm --filter @image-platform/gateway build",
"gateway:dev": "pnpm --filter @image-platform/gateway dev",
"gateway:start": "pnpm --filter @image-platform/gateway start",
"gateway:typecheck": "pnpm --filter @image-platform/gateway typecheck",
"queue:build": "pnpm --filter @image-platform/queue build",
"queue:typecheck": "pnpm --filter @image-platform/queue typecheck",
"storage:build": "pnpm --filter @image-platform/storage build",
"storage:typecheck": "pnpm --filter @image-platform/storage typecheck",
"worker:build": "pnpm db:build && pnpm queue:build && pnpm storage:build && pnpm --filter @image-platform/worker build",
"worker:dev": "pnpm db:build && pnpm queue:build && pnpm storage:build && pnpm --filter @image-platform/worker dev",
"worker:start": "pnpm --filter @image-platform/worker start",
"worker:typecheck": "pnpm --filter @image-platform/worker typecheck",
"infra:config": "docker compose -f infra/compose.dev.yml config",
"infra:up": "docker compose -f infra/compose.dev.yml up -d",
"infra:down": "docker compose -f infra/compose.dev.yml down",
"infra:logs": "docker compose -f infra/compose.dev.yml logs -f",
"check": "pnpm infra:config"
"check": "pnpm infra:config && pnpm db:typecheck && pnpm queue:typecheck && pnpm storage:typecheck && pnpm backend:typecheck && pnpm admin:typecheck && pnpm gateway:typecheck && pnpm worker:typecheck"
}
}

View File

@@ -0,0 +1,12 @@
import { defineConfig } from "drizzle-kit"
export default defineConfig({
dbCredentials: {
url: process.env.DATABASE_URL ?? "postgres://image:image-password@localhost:5433/image_platform",
},
dialect: "postgresql",
out: "./drizzle",
schema: "./src/schema.ts",
strict: true,
verbose: true,
})

View File

@@ -0,0 +1,72 @@
CREATE TYPE "public"."asset_status" AS ENUM('active', 'disabled', 'deleted');--> statement-breakpoint
CREATE TYPE "public"."requested_format" AS ENUM('auto', 'avif', 'webp', 'jpg', 'png');--> statement-breakpoint
CREATE TYPE "public"."variant_format" AS ENUM('avif', 'webp', 'jpg', 'png');--> statement-breakpoint
CREATE TYPE "public"."variant_status" AS ENUM('pending', 'processing', 'ready', 'failed');--> statement-breakpoint
CREATE TABLE "allowed_image_hosts" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"hostname" text NOT NULL,
"enabled" boolean DEFAULT true NOT NULL,
"description" text,
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
CREATE TABLE "image_asset_versions" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"asset_id" uuid NOT NULL,
"version" integer NOT NULL,
"source_url" text NOT NULL,
"source_host" text NOT NULL,
"source_hash" text NOT NULL,
"original_s3_key" text,
"width" integer,
"height" integer,
"content_type" text,
"size_bytes" bigint,
"created_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
CREATE TABLE "image_assets" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"public_id" text NOT NULL,
"current_version" integer DEFAULT 1 NOT NULL,
"status" "asset_status" DEFAULT 'active' NOT NULL,
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
CREATE TABLE "image_variants" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"asset_id" uuid NOT NULL,
"asset_version_id" uuid NOT NULL,
"asset_version" integer NOT NULL,
"preset" text NOT NULL,
"variant_hash" text NOT NULL,
"requested_format" "requested_format" DEFAULT 'auto' NOT NULL,
"format" "variant_format" NOT NULL,
"width" integer NOT NULL,
"height" integer,
"quality" integer NOT NULL,
"s3_key" text NOT NULL,
"content_type" text,
"etag" text,
"status" "variant_status" DEFAULT 'pending' NOT NULL,
"size_bytes" bigint,
"error" text,
"attempt_count" integer DEFAULT 0 NOT NULL,
"last_accessed_at" timestamp with time zone,
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
ALTER TABLE "image_asset_versions" ADD CONSTRAINT "image_asset_versions_asset_id_image_assets_id_fk" FOREIGN KEY ("asset_id") REFERENCES "public"."image_assets"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "image_variants" ADD CONSTRAINT "image_variants_asset_id_image_assets_id_fk" FOREIGN KEY ("asset_id") REFERENCES "public"."image_assets"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "image_variants" ADD CONSTRAINT "image_variants_asset_version_id_image_asset_versions_id_fk" FOREIGN KEY ("asset_version_id") REFERENCES "public"."image_asset_versions"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
CREATE UNIQUE INDEX "allowed_image_hosts_hostname_idx" ON "allowed_image_hosts" USING btree ("hostname");--> statement-breakpoint
CREATE UNIQUE INDEX "image_asset_versions_asset_version_idx" ON "image_asset_versions" USING btree ("asset_id","version");--> statement-breakpoint
CREATE INDEX "image_asset_versions_source_hash_idx" ON "image_asset_versions" USING btree ("source_hash");--> statement-breakpoint
CREATE UNIQUE INDEX "image_assets_public_id_idx" ON "image_assets" USING btree ("public_id");--> statement-breakpoint
CREATE UNIQUE INDEX "image_variants_lookup_idx" ON "image_variants" USING btree ("asset_id","asset_version","preset","width","quality","format");--> statement-breakpoint
CREATE UNIQUE INDEX "image_variants_s3_key_idx" ON "image_variants" USING btree ("s3_key");--> statement-breakpoint
CREATE UNIQUE INDEX "image_variants_variant_hash_idx" ON "image_variants" USING btree ("variant_hash");--> statement-breakpoint
CREATE INDEX "image_variants_status_idx" ON "image_variants" USING btree ("status");

View File

@@ -0,0 +1,604 @@
{
"id": "72292622-d326-46fe-8e6a-90096c7e6634",
"prevId": "00000000-0000-0000-0000-000000000000",
"version": "7",
"dialect": "postgresql",
"tables": {
"public.allowed_image_hosts": {
"name": "allowed_image_hosts",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"hostname": {
"name": "hostname",
"type": "text",
"primaryKey": false,
"notNull": true
},
"enabled": {
"name": "enabled",
"type": "boolean",
"primaryKey": false,
"notNull": true,
"default": true
},
"description": {
"name": "description",
"type": "text",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"allowed_image_hosts_hostname_idx": {
"name": "allowed_image_hosts_hostname_idx",
"columns": [
{
"expression": "hostname",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.image_asset_versions": {
"name": "image_asset_versions",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"asset_id": {
"name": "asset_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"version": {
"name": "version",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"source_url": {
"name": "source_url",
"type": "text",
"primaryKey": false,
"notNull": true
},
"source_host": {
"name": "source_host",
"type": "text",
"primaryKey": false,
"notNull": true
},
"source_hash": {
"name": "source_hash",
"type": "text",
"primaryKey": false,
"notNull": true
},
"original_s3_key": {
"name": "original_s3_key",
"type": "text",
"primaryKey": false,
"notNull": false
},
"width": {
"name": "width",
"type": "integer",
"primaryKey": false,
"notNull": false
},
"height": {
"name": "height",
"type": "integer",
"primaryKey": false,
"notNull": false
},
"content_type": {
"name": "content_type",
"type": "text",
"primaryKey": false,
"notNull": false
},
"size_bytes": {
"name": "size_bytes",
"type": "bigint",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"image_asset_versions_asset_version_idx": {
"name": "image_asset_versions_asset_version_idx",
"columns": [
{
"expression": "asset_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "version",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"image_asset_versions_source_hash_idx": {
"name": "image_asset_versions_source_hash_idx",
"columns": [
{
"expression": "source_hash",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"image_asset_versions_asset_id_image_assets_id_fk": {
"name": "image_asset_versions_asset_id_image_assets_id_fk",
"tableFrom": "image_asset_versions",
"tableTo": "image_assets",
"columnsFrom": [
"asset_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.image_assets": {
"name": "image_assets",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"public_id": {
"name": "public_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"current_version": {
"name": "current_version",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 1
},
"status": {
"name": "status",
"type": "asset_status",
"typeSchema": "public",
"primaryKey": false,
"notNull": true,
"default": "'active'"
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"image_assets_public_id_idx": {
"name": "image_assets_public_id_idx",
"columns": [
{
"expression": "public_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.image_variants": {
"name": "image_variants",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"asset_id": {
"name": "asset_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"asset_version_id": {
"name": "asset_version_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"asset_version": {
"name": "asset_version",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"preset": {
"name": "preset",
"type": "text",
"primaryKey": false,
"notNull": true
},
"variant_hash": {
"name": "variant_hash",
"type": "text",
"primaryKey": false,
"notNull": true
},
"requested_format": {
"name": "requested_format",
"type": "requested_format",
"typeSchema": "public",
"primaryKey": false,
"notNull": true,
"default": "'auto'"
},
"format": {
"name": "format",
"type": "variant_format",
"typeSchema": "public",
"primaryKey": false,
"notNull": true
},
"width": {
"name": "width",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"height": {
"name": "height",
"type": "integer",
"primaryKey": false,
"notNull": false
},
"quality": {
"name": "quality",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"s3_key": {
"name": "s3_key",
"type": "text",
"primaryKey": false,
"notNull": true
},
"content_type": {
"name": "content_type",
"type": "text",
"primaryKey": false,
"notNull": false
},
"etag": {
"name": "etag",
"type": "text",
"primaryKey": false,
"notNull": false
},
"status": {
"name": "status",
"type": "variant_status",
"typeSchema": "public",
"primaryKey": false,
"notNull": true,
"default": "'pending'"
},
"size_bytes": {
"name": "size_bytes",
"type": "bigint",
"primaryKey": false,
"notNull": false
},
"error": {
"name": "error",
"type": "text",
"primaryKey": false,
"notNull": false
},
"attempt_count": {
"name": "attempt_count",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 0
},
"last_accessed_at": {
"name": "last_accessed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"image_variants_lookup_idx": {
"name": "image_variants_lookup_idx",
"columns": [
{
"expression": "asset_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "asset_version",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "preset",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "width",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "quality",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "format",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"image_variants_s3_key_idx": {
"name": "image_variants_s3_key_idx",
"columns": [
{
"expression": "s3_key",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"image_variants_variant_hash_idx": {
"name": "image_variants_variant_hash_idx",
"columns": [
{
"expression": "variant_hash",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"image_variants_status_idx": {
"name": "image_variants_status_idx",
"columns": [
{
"expression": "status",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"image_variants_asset_id_image_assets_id_fk": {
"name": "image_variants_asset_id_image_assets_id_fk",
"tableFrom": "image_variants",
"tableTo": "image_assets",
"columnsFrom": [
"asset_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"image_variants_asset_version_id_image_asset_versions_id_fk": {
"name": "image_variants_asset_version_id_image_asset_versions_id_fk",
"tableFrom": "image_variants",
"tableTo": "image_asset_versions",
"columnsFrom": [
"asset_version_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
}
},
"enums": {
"public.asset_status": {
"name": "asset_status",
"schema": "public",
"values": [
"active",
"disabled",
"deleted"
]
},
"public.requested_format": {
"name": "requested_format",
"schema": "public",
"values": [
"auto",
"avif",
"webp",
"jpg",
"png"
]
},
"public.variant_format": {
"name": "variant_format",
"schema": "public",
"values": [
"avif",
"webp",
"jpg",
"png"
]
},
"public.variant_status": {
"name": "variant_status",
"schema": "public",
"values": [
"pending",
"processing",
"ready",
"failed"
]
}
},
"schemas": {},
"sequences": {},
"roles": {},
"policies": {},
"views": {},
"_meta": {
"columns": {},
"schemas": {},
"tables": {}
}
}

View File

@@ -0,0 +1,13 @@
{
"version": "7",
"dialect": "postgresql",
"entries": [
{
"idx": 0,
"version": "7",
"when": 1777963363578,
"tag": "0000_calm_magdalene",
"breakpoints": true
}
]
}

View File

@@ -0,0 +1,30 @@
{
"name": "@image-platform/database",
"version": "0.1.0",
"private": true,
"type": "module",
"exports": {
".": {
"types": "./src/index.ts",
"default": "./dist/index.js"
}
},
"types": "./src/index.ts",
"scripts": {
"build": "tsc -p tsconfig.build.json",
"db:generate": "drizzle-kit generate --config drizzle.config.ts",
"db:migrate": "drizzle-kit migrate --config drizzle.config.ts",
"db:studio": "drizzle-kit studio --config drizzle.config.ts",
"typecheck": "tsc --noEmit -p tsconfig.json"
},
"dependencies": {
"drizzle-orm": "^0.45.2",
"pg": "^8.20.0"
},
"devDependencies": {
"@types/node": "^25.6.0",
"@types/pg": "^8.20.0",
"drizzle-kit": "^0.31.10",
"typescript": "^6.0.3"
}
}

View File

@@ -0,0 +1,21 @@
import { drizzle } from "drizzle-orm/node-postgres"
import pg from "pg"
import * as schema from "./schema.js"
const { Pool } = pg
export type DatabasePool = pg.Pool
export type Database = ReturnType<typeof createDatabase>
export function createDatabasePool(databaseUrl = process.env.DATABASE_URL) {
if (!databaseUrl) {
throw new Error("DATABASE_URL is required")
}
return new Pool({ connectionString: databaseUrl })
}
export function createDatabase(pool: DatabasePool) {
return drizzle(pool, { schema })
}

View File

@@ -0,0 +1,2 @@
export { createDatabase, createDatabasePool } from "./client.js"
export * from "./schema.js"

View File

@@ -0,0 +1,113 @@
import {
bigint,
boolean,
index,
integer,
pgEnum,
pgTable,
text,
timestamp,
uniqueIndex,
uuid,
} from "drizzle-orm/pg-core"
export const assetStatusEnum = pgEnum("asset_status", ["active", "disabled", "deleted"])
export const variantFormatEnum = pgEnum("variant_format", ["avif", "webp", "jpg", "png"])
export const requestedFormatEnum = pgEnum("requested_format", ["auto", "avif", "webp", "jpg", "png"])
export const variantStatusEnum = pgEnum("variant_status", ["pending", "processing", "ready", "failed"])
const timestamps = {
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(),
}
export const allowedImageHosts = pgTable(
"allowed_image_hosts",
{
id: uuid("id").primaryKey().defaultRandom(),
hostname: text("hostname").notNull(),
enabled: boolean("enabled").notNull().default(true),
description: text("description"),
...timestamps,
},
(table) => [uniqueIndex("allowed_image_hosts_hostname_idx").on(table.hostname)],
)
export const imageAssets = pgTable(
"image_assets",
{
id: uuid("id").primaryKey().defaultRandom(),
publicId: text("public_id").notNull(),
currentVersion: integer("current_version").notNull().default(1),
status: assetStatusEnum("status").notNull().default("active"),
...timestamps,
},
(table) => [uniqueIndex("image_assets_public_id_idx").on(table.publicId)],
)
export const imageAssetVersions = pgTable(
"image_asset_versions",
{
id: uuid("id").primaryKey().defaultRandom(),
assetId: uuid("asset_id")
.notNull()
.references(() => imageAssets.id, { onDelete: "cascade" }),
version: integer("version").notNull(),
sourceUrl: text("source_url").notNull(),
sourceHost: text("source_host").notNull(),
sourceHash: text("source_hash").notNull(),
originalS3Key: text("original_s3_key"),
width: integer("width"),
height: integer("height"),
contentType: text("content_type"),
sizeBytes: bigint("size_bytes", { mode: "number" }),
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
},
(table) => [
uniqueIndex("image_asset_versions_asset_version_idx").on(table.assetId, table.version),
index("image_asset_versions_source_hash_idx").on(table.sourceHash),
],
)
export const imageVariants = pgTable(
"image_variants",
{
id: uuid("id").primaryKey().defaultRandom(),
assetId: uuid("asset_id")
.notNull()
.references(() => imageAssets.id, { onDelete: "cascade" }),
assetVersionId: uuid("asset_version_id")
.notNull()
.references(() => imageAssetVersions.id, { onDelete: "cascade" }),
assetVersion: integer("asset_version").notNull(),
preset: text("preset").notNull(),
variantHash: text("variant_hash").notNull(),
requestedFormat: requestedFormatEnum("requested_format").notNull().default("auto"),
format: variantFormatEnum("format").notNull(),
width: integer("width").notNull(),
height: integer("height"),
quality: integer("quality").notNull(),
s3Key: text("s3_key").notNull(),
contentType: text("content_type"),
etag: text("etag"),
status: variantStatusEnum("status").notNull().default("pending"),
sizeBytes: bigint("size_bytes", { mode: "number" }),
error: text("error"),
attemptCount: integer("attempt_count").notNull().default(0),
lastAccessedAt: timestamp("last_accessed_at", { withTimezone: true }),
...timestamps,
},
(table) => [
uniqueIndex("image_variants_lookup_idx").on(
table.assetId,
table.assetVersion,
table.preset,
table.width,
table.quality,
table.format,
),
uniqueIndex("image_variants_s3_key_idx").on(table.s3Key),
uniqueIndex("image_variants_variant_hash_idx").on(table.variantHash),
index("image_variants_status_idx").on(table.status),
],
)

View File

@@ -0,0 +1,7 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"noEmit": false
},
"exclude": ["dist", "node_modules", "drizzle.config.ts"]
}

View File

@@ -0,0 +1,21 @@
{
"compilerOptions": {
"allowSyntheticDefaultImports": true,
"declaration": true,
"esModuleInterop": true,
"forceConsistentCasingInFileNames": true,
"lib": ["ES2023"],
"module": "NodeNext",
"moduleResolution": "NodeNext",
"noUncheckedIndexedAccess": true,
"outDir": "./dist",
"rootDir": "./src",
"skipLibCheck": true,
"sourceMap": true,
"strict": true,
"target": "ES2023",
"types": ["node"]
},
"include": ["src/**/*.ts"],
"exclude": ["dist", "node_modules"]
}

View File

@@ -0,0 +1,21 @@
{
"name": "@image-platform/queue",
"version": "0.1.0",
"private": true,
"type": "module",
"exports": {
".": {
"types": "./src/index.ts",
"default": "./dist/index.js"
}
},
"types": "./src/index.ts",
"scripts": {
"build": "tsc -p tsconfig.build.json",
"typecheck": "tsc --noEmit -p tsconfig.json"
},
"devDependencies": {
"@types/node": "^25.6.0",
"typescript": "^6.0.3"
}
}

View File

@@ -0,0 +1,2 @@
export * from "./jobs.js"
export * from "./topology.js"

View File

@@ -0,0 +1,37 @@
export type GenerateVariantJob = {
jobId: string
variantId: string
}
export function parseGenerateVariantJobBuffer(buffer: Buffer): GenerateVariantJob {
const value = JSON.parse(buffer.toString("utf8")) as unknown
return parseGenerateVariantJob(value)
}
export function parseGenerateVariantJob(value: unknown): GenerateVariantJob {
if (!isRecord(value)) {
throw new Error("generate variant job must be a JSON object")
}
if (!isNonEmptyString(value.jobId)) {
throw new Error("generate variant job must include jobId")
}
if (!isNonEmptyString(value.variantId)) {
throw new Error("generate variant job must include variantId")
}
return {
jobId: value.jobId,
variantId: value.variantId,
}
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value)
}
function isNonEmptyString(value: unknown): value is string {
return typeof value === "string" && value.trim().length > 0
}

View File

@@ -0,0 +1,33 @@
export type QueueTopology = {
generateVariantDeadLetterQueue: string
generateVariantDeadLetterRoutingKey: string
generateVariantQueue: string
generateVariantRoutingKey: string
jobsDeadLetterExchange: string
jobsExchange: string
}
export const DEFAULT_QUEUE_TOPOLOGY: QueueTopology = {
generateVariantDeadLetterQueue: "image.generate-variant.dlq",
generateVariantDeadLetterRoutingKey: "image.generate-variant.dlq",
generateVariantQueue: "image.generate-variant",
generateVariantRoutingKey: "image.generate-variant",
jobsDeadLetterExchange: "image-platform.jobs.dlx",
jobsExchange: "image-platform.jobs",
}
export function loadQueueTopologyFromEnv(env: NodeJS.ProcessEnv = process.env): QueueTopology {
const generateVariantQueue = env.RABBITMQ_GENERATE_VARIANT_QUEUE ?? DEFAULT_QUEUE_TOPOLOGY.generateVariantQueue
const generateVariantDeadLetterQueue =
env.RABBITMQ_GENERATE_VARIANT_DLQ ?? DEFAULT_QUEUE_TOPOLOGY.generateVariantDeadLetterQueue
return {
generateVariantDeadLetterQueue,
generateVariantDeadLetterRoutingKey:
env.RABBITMQ_GENERATE_VARIANT_DLQ_ROUTING_KEY ?? generateVariantDeadLetterQueue,
generateVariantQueue,
generateVariantRoutingKey: env.RABBITMQ_GENERATE_VARIANT_ROUTING_KEY ?? generateVariantQueue,
jobsDeadLetterExchange: env.RABBITMQ_GENERATE_VARIANT_DLX ?? DEFAULT_QUEUE_TOPOLOGY.jobsDeadLetterExchange,
jobsExchange: env.RABBITMQ_JOBS_EXCHANGE ?? DEFAULT_QUEUE_TOPOLOGY.jobsExchange,
}
}

View File

@@ -0,0 +1,7 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"noEmit": false
},
"exclude": ["dist", "node_modules", "**/*.spec.ts"]
}

View File

@@ -0,0 +1,21 @@
{
"compilerOptions": {
"allowSyntheticDefaultImports": true,
"declaration": true,
"esModuleInterop": true,
"forceConsistentCasingInFileNames": true,
"lib": ["ES2023"],
"module": "NodeNext",
"moduleResolution": "NodeNext",
"noUncheckedIndexedAccess": true,
"outDir": "./dist",
"rootDir": "./src",
"skipLibCheck": true,
"sourceMap": true,
"strict": true,
"target": "ES2023",
"types": ["node"]
},
"include": ["src/**/*.ts"],
"exclude": ["dist", "node_modules"]
}

View File

@@ -0,0 +1,24 @@
{
"name": "@image-platform/storage",
"version": "0.1.0",
"private": true,
"type": "module",
"exports": {
".": {
"types": "./src/index.ts",
"default": "./dist/index.js"
}
},
"types": "./src/index.ts",
"scripts": {
"build": "tsc -p tsconfig.build.json",
"typecheck": "tsc --noEmit -p tsconfig.json"
},
"dependencies": {
"@aws-sdk/client-s3": "^3.1042.0"
},
"devDependencies": {
"@types/node": "^25.6.0",
"typescript": "^6.0.3"
}
}

View File

@@ -0,0 +1,20 @@
import { S3Client, type S3ClientConfig } from "@aws-sdk/client-s3"
import type { StorageConfig } from "./config.js"
export function createS3Client(config: StorageConfig) {
const clientConfig: S3ClientConfig = {
endpoint: config.endpoint,
forcePathStyle: config.forcePathStyle,
region: config.region,
}
if (config.accessKeyId && config.secretAccessKey) {
clientConfig.credentials = {
accessKeyId: config.accessKeyId,
secretAccessKey: config.secretAccessKey,
}
}
return new S3Client(clientConfig)
}

View File

@@ -0,0 +1,31 @@
export type StorageConfig = {
accessKeyId?: string
bucket: string
endpoint?: string
forcePathStyle: boolean
region: string
secretAccessKey?: string
}
export function loadStorageConfigFromEnv(env: NodeJS.ProcessEnv = process.env): StorageConfig {
return {
accessKeyId: normalizeOptionalString(env.S3_ACCESS_KEY_ID),
bucket: env.S3_BUCKET ?? "image-platform",
endpoint: normalizeOptionalString(env.S3_ENDPOINT),
forcePathStyle: parseBoolean(env.S3_FORCE_PATH_STYLE, true),
region: env.S3_REGION ?? "us-east-1",
secretAccessKey: normalizeOptionalString(env.S3_SECRET_ACCESS_KEY),
}
}
function normalizeOptionalString(value: string | undefined) {
return value && value.trim().length > 0 ? value : undefined
}
function parseBoolean(value: string | undefined, fallback: boolean) {
if (value === undefined) {
return fallback
}
return ["1", "true", "yes"].includes(value.toLowerCase())
}

View File

@@ -0,0 +1,3 @@
export * from "./client.js"
export * from "./config.js"
export * from "./keys.js"

View File

@@ -0,0 +1,40 @@
export type VariantFormat = "avif" | "jpg" | "png" | "webp"
export type OriginalImageKeyInput = {
assetId: string
version: number
}
export type VariantImageKeyInput = OriginalImageKeyInput & {
format: VariantFormat
variantHash: string
}
export function buildOriginalImageKey(input: OriginalImageKeyInput) {
return `originals/${safeSegment(input.assetId, "assetId")}/v${safeVersion(input.version)}/source`
}
export function buildVariantImageKey(input: VariantImageKeyInput) {
return [
"variants",
safeSegment(input.assetId, "assetId"),
`v${safeVersion(input.version)}`,
`${safeSegment(input.variantHash, "variantHash")}.${input.format}`,
].join("/")
}
function safeSegment(value: string, name: string) {
if (value.length === 0 || value.includes("/")) {
throw new Error(`${name} must be a non-empty S3 key segment`)
}
return value
}
function safeVersion(value: number) {
if (!Number.isSafeInteger(value) || value < 1) {
throw new Error("version must be a positive integer")
}
return value
}

View File

@@ -0,0 +1,7 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"noEmit": false
},
"exclude": ["dist", "node_modules", "**/*.spec.ts"]
}

View File

@@ -0,0 +1,21 @@
{
"compilerOptions": {
"allowSyntheticDefaultImports": true,
"declaration": true,
"esModuleInterop": true,
"forceConsistentCasingInFileNames": true,
"lib": ["ES2023"],
"module": "NodeNext",
"moduleResolution": "NodeNext",
"noUncheckedIndexedAccess": true,
"outDir": "./dist",
"rootDir": "./src",
"skipLibCheck": true,
"sourceMap": true,
"strict": true,
"target": "ES2023",
"types": ["node"]
},
"include": ["src/**/*.ts"],
"exclude": ["dist", "node_modules"]
}

5979
pnpm-lock.yaml generated

File diff suppressed because it is too large Load Diff