Media Understanding

POST /api/v1/media/understand labels or extracts structured information from images, audio, and video. You send the instructions, the media, and a JSON Schema; the response is a JSON object that conforms to your schema — closed label sets, scores, timestamps, transcribed fields, whatever you define. There is no free-text mode: this endpoint is built for tagging and extraction pipelines that need machine-checkable output.

Beta: available to any account with an active paid subscription. Synchronous; a frontier multimodal model with automatic fallback runs the request — you never pick or see the model.

Request

{
  "prompt": "Classify the product shown and estimate the shot type.",
  "inputs": [
    {"url": "https://example.com/product.jpg"},
    {"asset_id": "7241058823145623552"}
  ],
  "schema": {
    "type": "object",
    "properties": {
      "category": {"type": "string", "enum": ["apparel", "electronics", "food", "other"]},
      "shot_type": {"type": "string", "enum": ["product_only", "on_model", "lifestyle"]},
      "confidence": {"type": "number", "minimum": 0, "maximum": 1}
    },
    "required": ["category", "shot_type", "confidence"]
  }
}

• inputs: 1–8 media items, each an asset_id, a public url, or inline_data; image, audio, and video may be mixed. One-off url / inline_data inputs are not added to your asset library.
• schema: JSON Schema with a top-level object. Keep it flat; use enum for closed label sets and a description per field — the model reads them. Up to 20 KB.
• prompt: up to 5,000 characters. Say what each field means when it is not obvious.

curl -s -X POST -H "X-API-Key: $VISIONSTORY_API_KEY" -H "Content-Type: application/json" -d @request.json https://openapi.visionstory.ai/api/v1/media/understand

Response

{
  "data": {
    "output": {"category": "apparel", "shot_type": "on_model", "confidence": 0.92},
    "usage": {"input_tokens": 1416, "output_tokens": 58},
    "cost_credit": 1
  },
  "message": "success",
  "server_time": "2026-09-03T08:00:00Z"
}

output is validated against your schema before it is returned. If the model completes but its answer cannot be made to conform (after one retry and a fallback), the call fails with error.code 1 — the upstream usage that was actually consumed is still billed, because it was your prompt and schema that ran. Provider errors, timeouts, and content rejections are never charged.

Billing

Cost follows the upstream model's token usage at 1 credit per $0.10, rounded up to a whole credit per call, minimum 1 credit. usage shows the tokens you were billed on, summed over every upstream attempt the request needed. Rough sizes: an image is about 1,100 input tokens, audio about 32 tokens per second, video about 300 tokens per second at default resolution, and the output (including the model's reasoning) is priced at the higher output rate. A single image with a short schema is 1 credit; a 3-minute video is typically 1–2 credits. A balance pre-check runs before the call using these estimates, so keep a small margin of paid credits.

Limits

Content that the upstream provider refuses is rejected with error.code 37100 and is not charged. Inputs are sent to the upstream model provider for the duration of the request and are not retained by VisionStory beyond it.

Errors