Media Understanding/Understand media (structured)

Understand media (structured)

POST/api/v1/media/understand

Label or extract structured information from images, audio, and video with a frontier multimodal model. You supply the instructions (prompt), the media (inputs), and a JSON schema; the response output conforms to that schema — closed label sets, scores, timestamps, transcribed fields, whatever you define. Synchronous. Billing follows the upstream model's token usage (input media and prompt, plus output including reasoning) at 1 credit per $0.10, rounded up per call, charged only on success. Per-key concurrency is limited during beta; requests beyond the limit are rejected, not queued.

Headers

X-API-KeystringRequired

Your VisionStory API key (sk-vs-...), kept server-side. Create one at OpenApi (Pro plan and up).

Request body

promptstringRequired

What to extract or label. Describe the task and, if helpful, the meaning of each schema field; the model sees the media in inputs in the given order.

inputsarray of MediaRefRequired

1-8 media items (image / audio / video), each as asset_id, public url, or inline_data. Mixed kinds are fine. Not stored beyond the request.

Show 3 propertiesHide 3 properties

asset_idstring | nullOptional

Asset ID from POST /api/v1/asset; use for materials reused across requests.

urlstring | nullOptional

Publicly accessible media URL for one-off use; not added to your asset library.

inline_dataInlineDataModel | nullOptional

Inline base64 media data for one-off use; not added to your asset library. Images: image/jpeg, image/jpg, image/png, image/webp, image/bmp, image/tiff, image/gif; audio: audio/wav, audio/x-wav, audio/wave, audio/mpeg, audio/mp3; video: video/mp4, video/quicktime, video/mov.

Show 2 propertiesHide 2 properties

mime_typestringRequired

MIME type of the inline data; the gateway uses it to tell image / audio / video apart. The accepted set depends on the endpoint — see the field carrying this object. Talking video accepts audio ['audio/avi', 'audio/mpeg', 'audio/mp3', 'audio/mp4', 'audio/m4a', 'audio/wav'] and images ['image/jpeg', 'image/jpg', 'image/png', 'image/webp', 'image/heic'].

datastringRequired

The file's raw bytes encoded as a base64 string (no data: URI prefix).

schemaobjectRequired

JSON Schema (draft 2020-12 subset) the output must conform to. Top level must be an object. Keep it flat and explicit — enums for closed label sets, description on each field.

Response

200Successful Response

Successful calls return a standard envelope: the endpoint payload under data (its fields are documented below), plus a message string ("success") and an ISO 8601 server_time.

Response fields (data)

outputobjectRequired

The structured result, conforming to the request schema.

usageMediaUnderstandUsageRequired

Upstream token usage this call was billed on.

Show 2 propertiesHide 2 properties

input_tokensintegerOptionalDefault 0

Tokens consumed by the prompt and media inputs.

output_tokensintegerOptionalDefault 0

Tokens generated (including the model's reasoning).

cost_creditintegerRequired

Credits charged for this call (usage priced at upstream rates, rounded up to a whole credit, minimum 1).

Errors

All error responses share one JSON envelope: an error object with a numeric code, a human-readable message, an optional details string, and an optional hint giving an actionable next step (handy for AI agents).

errorErrorDetailRequired

Error payload returned with every non-2xx response. Present only on failure; successful calls use the standard success envelope instead.

Show 4 propertiesHide 4 properties

codeintegerRequired

Machine-readable error code. Mirrors the HTTP status for transport-level failures (e.g. 401, 404, 422, 500) and may carry a business-specific code otherwise.

messagestringRequired

Human-readable explanation of what went wrong. Safe to log or surface to end users; not localized.

detailsstring | nullOptional

Optional structured detail about the failure, e.g. a JSON string of per-field validation errors on a 422. Absent when there is nothing extra to report.

hintstring | nullOptional

Actionable next step for resolving the error, written for both humans and AI agents (e.g. how to fix the request, or where to obtain an API key). May be absent.

Request

curl -X POST 'https://openapi.visionstory.ai/api/v1/media/understand' \
  -H 'X-API-Key: sk-vs-xxxxxxxxxxxxxxxxxxx' \
  -H 'Content-Type: application/json' \
  -d '{
  "prompt": "Identify the main subject in the image.",
  "inputs": [
    {
      "url": "https://example.com/cat.jpg"
    }
  ],
  "schema": {
    "type": "object",
    "properties": {
      "subject": {
        "type": "string",
        "description": "Main subject in the image."
      }
    },
    "required": [
      "subject"
    ],
    "additionalProperties": false
  }
}'

import requests

url = "https://openapi.visionstory.ai/api/v1/media/understand"
headers = {
    "X-API-Key": "sk-vs-xxxxxxxxxxxxxxxxxxx",
    "Content-Type": "application/json",
}
payload = {
  "prompt": "Identify the main subject in the image.",
  "inputs": [
    {
      "url": "https://example.com/cat.jpg"
    }
  ],
  "schema": {
    "type": "object",
    "properties": {
      "subject": {
        "type": "string",
        "description": "Main subject in the image."
      }
    },
    "required": [
      "subject"
    ],
    "additionalProperties": False
  }
}
response = requests.post(url, headers=headers, json=payload)
print(response.json())

const response = await fetch("https://openapi.visionstory.ai/api/v1/media/understand", {
  method: "POST",
  headers: {
    "X-API-Key": "sk-vs-xxxxxxxxxxxxxxxxxxx",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "prompt": "Identify the main subject in the image.",
    "inputs": [
      {
        "url": "https://example.com/cat.jpg"
      }
    ],
    "schema": {
      "type": "object",
      "properties": {
        "subject": {
          "type": "string",
          "description": "Main subject in the image."
        }
      },
      "required": [
        "subject"
      ],
      "additionalProperties": false
    }
  }),
});
const data = await response.json();
console.log(data);

package main

import (
	"bytes"
	"fmt"
	"io"
	"net/http"
)

func main() {
	payload := []byte(`{
  "prompt": "Identify the main subject in the image.",
  "inputs": [
    {
      "url": "https://example.com/cat.jpg"
    }
  ],
  "schema": {
    "type": "object",
    "properties": {
      "subject": {
        "type": "string",
        "description": "Main subject in the image."
      }
    },
    "required": [
      "subject"
    ],
    "additionalProperties": false
  }
}`)
	req, _ := http.NewRequest("POST", "https://openapi.visionstory.ai/api/v1/media/understand", bytes.NewBuffer(payload))
	req.Header.Set("X-API-Key", "sk-vs-xxxxxxxxxxxxxxxxxxx")
	req.Header.Set("Content-Type", "application/json")
	resp, _ := http.DefaultClient.Do(req)
	defer resp.Body.Close()
	body, _ := io.ReadAll(resp.Body)
	fmt.Println(string(body))
}

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

HttpClient client = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder()
    .uri(URI.create("https://openapi.visionstory.ai/api/v1/media/understand"))
    .header("X-API-Key", "sk-vs-xxxxxxxxxxxxxxxxxxx")
    .header("Content-Type", "application/json")
    .method("POST", HttpRequest.BodyPublishers.ofString("""
        {
          "prompt": "Identify the main subject in the image.",
          "inputs": [
            {
              "url": "https://example.com/cat.jpg"
            }
          ],
          "schema": {
            "type": "object",
            "properties": {
              "subject": {
                "type": "string",
                "description": "Main subject in the image."
              }
            },
            "required": [
              "subject"
            ],
            "additionalProperties": false
          }
        }"""))
    .build();
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println(response.body());

<?php
$ch = curl_init("https://openapi.visionstory.ai/api/v1/media/understand");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_HTTPHEADER, [
    "X-API-Key: sk-vs-xxxxxxxxxxxxxxxxxxx",
    "Content-Type: application/json",
]);
$payload = <<<'JSON'
{
  "prompt": "Identify the main subject in the image.",
  "inputs": [
    {
      "url": "https://example.com/cat.jpg"
    }
  ],
  "schema": {
    "type": "object",
    "properties": {
      "subject": {
        "type": "string",
        "description": "Main subject in the image."
      }
    },
    "required": [
      "subject"
    ],
    "additionalProperties": false
  }
}
JSON;
curl_setopt($ch, CURLOPT_POSTFIELDS, $payload);
$response = curl_exec($ch);
curl_close($ch);
echo $response;

require "net/http"
require "uri"

uri = URI("https://openapi.visionstory.ai/api/v1/media/understand")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
request = Net::HTTP::Post.new(uri)
request["X-API-Key"] = "sk-vs-xxxxxxxxxxxxxxxxxxx"
request["Content-Type"] = "application/json"
request.body = <<~JSON
  {
    "prompt": "Identify the main subject in the image.",
    "inputs": [
      {
        "url": "https://example.com/cat.jpg"
      }
    ],
    "schema": {
      "type": "object",
      "properties": {
        "subject": {
          "type": "string",
          "description": "Main subject in the image."
        }
      },
      "required": [
        "subject"
      ],
      "additionalProperties": false
    }
  }
  JSON
response = http.request(request)
puts response.body

import Foundation

var request = URLRequest(url: URL(string: "https://openapi.visionstory.ai/api/v1/media/understand")!)
request.httpMethod = "POST"
request.setValue("sk-vs-xxxxxxxxxxxxxxxxxxx", forHTTPHeaderField: "X-API-Key")
request.setValue("application/json", forHTTPHeaderField: "Content-Type")
request.httpBody = """
{
  "prompt": "Identify the main subject in the image.",
  "inputs": [
    {
      "url": "https://example.com/cat.jpg"
    }
  ],
  "schema": {
    "type": "object",
    "properties": {
      "subject": {
        "type": "string",
        "description": "Main subject in the image."
      }
    },
    "required": [
      "subject"
    ],
    "additionalProperties": false
  }
}
""".data(using: .utf8)
let (data, _) = try await URLSession.shared.data(for: request)
print(String(data: data, encoding: .utf8) ?? "")

using System;
using System.Net.Http;
using System.Text;

var client = new HttpClient();
var request = new HttpRequestMessage(HttpMethod.Post, "https://openapi.visionstory.ai/api/v1/media/understand");
request.Headers.Add("X-API-Key", "sk-vs-xxxxxxxxxxxxxxxxxxx");
var payload = """
{
  "prompt": "Identify the main subject in the image.",
  "inputs": [
    {
      "url": "https://example.com/cat.jpg"
    }
  ],
  "schema": {
    "type": "object",
    "properties": {
      "subject": {
        "type": "string",
        "description": "Main subject in the image."
      }
    },
    "required": [
      "subject"
    ],
    "additionalProperties": false
  }
}
""";
request.Content = new StringContent(payload, Encoding.UTF8, "application/json");
var response = await client.SendAsync(request);
Console.WriteLine(await response.Content.ReadAsStringAsync());

Response

200

{
  "data": {
    "output": {
      "subject": "cat"
    },
    "usage": {
      "input_tokens": 120,
      "output_tokens": 12
    },
    "cost_credit": 1
  },
  "message": "success",
  "server_time": "2026-09-04T00:00:00Z"
}