Getting Started

Introduction

Warpstack is the Data Acquisition Cloud — built for alternative datasets. Add new data sources in hours, stop maintaining scrapers, and ship with full audit trails from day one.

The full stack, handled for you:

• Acquisition: AI agents connect to any source—websites, APIs, databases—and extract the data you need
• Structuring: Raw data is automatically cleaned, validated, and transformed to your schema
• Hosting: Your data lives in our cloud with low-latency access and automatic backups
• Serving: Production-ready REST APIs with filtering, sorting, and pagination—ready from day one
• Monitoring: Complete audit trails, observability, and adaptive handling when sources change

Our AI agents use vision to "see" pages like a human—reading charts, handling dynamic JavaScript, and adapting to layout changes without breaking. But Warpstack isn't just about extraction. It's about replacing your entire data stack with a single platform.

How It Works

From description to production API in minutes:

1. Describe your data: Tell us what you need in plain English—no code, no selectors
2. We generate your schema: AI creates a structured OpenAPI schema from your description
3. Deploy: Click "Deploy Warp" and we handle acquisition, structuring, and hosting
4. Access your API: Your data is immediately available via a production REST endpoint
5. Set a schedule: Keep data fresh with automatic hourly, daily, or weekly refreshes

Why Warpstack?

Alternative data vendors spend 30-60% of engineering time maintaining scrapers. Warpstack eliminates that burden:

• Ship more datasets: Add new sources in hours, not months—expand coverage faster
• Zero maintenance: AI agents adapt when sites change—no more 3am pages
• Built-in compliance: Full audit trails for every record—answer "where did this data come from?" in seconds
• Production APIs included: Every dataset is a REST API with filtering, sorting, and pagination
• Managed hosting: We host your data with automatic backups and retention policies

Current Limitations

Warpstack works best with public data sources. Current limitations:

• Login-protected content: Cannot authenticate to gated pages (coming soon)
• CAPTCHA challenges: Cannot reliably solve CAPTCHAs
• Real-time streaming: Designed for periodic acquisition, not sub-second updates
• File processing: Cannot download or parse PDFs, spreadsheets, or documents

Core Concepts

Understanding these key concepts will help you get the most out of Warpstack.

Warp

A warp is an autonomous data extraction and serving job. Each warp is configured with:

• Target URL: The website page to extract data from
• Prompt: Natural language description of what data to extract
• Schema: OpenAPI definition of the data structure
• Schedule: How often to run (manual, hourly, daily, weekly)
• Intelligence tier: Which AI model powers the extraction

Schema

The schema is the most important part of your warp. It serves two purposes:

• API contract: Defines the JSON structure your API returns
• Extraction instructions: Field descriptions guide the agent on what to extract

Schemas are written in OpenAPI 3.1.0 format and follow Structured Outputs rules. The quality of your prompt directly determines the quality of your schema, which determines extraction accuracy.

Stargate API

Stargate is the data serving layer. Once your warp extracts data, it's instantly available via your unique API endpoint:

https://{your-warp-id}.stargate.warpstack.ai

Stargate supports SQL-like filtering, sorting, pagination, column selection, and multiple output formats (JSON, CSV). Endpoints are public by default, but can be secured with access tokens.

Warp States

Warps move through different states during their lifecycle:

• DRAFT: Being configured. Schema is editable, not yet deployed.
• DEPLOYING: Setting up infrastructure for the first extraction.
• ACTIVE: Deployed and serving data. Schema is locked.
• RUNNING: Currently executing an extraction job.
• SCHEDULED: Active with a recurring schedule configured.
• PAUSED: Temporarily stopped. API still serves existing data.

Intelligence Tiers

Choose the AI model that powers your extraction:

• V1-Lite: For simple, straightforward pages.
• V1-Base: Recommended. For complex sites with dynamic content.

See Intelligence Tiers for details.

Quickstart

Get your first warp running in under 5 minutes. This guide walks you through the essential steps.

Step 1: Create a Warp

From your dashboard, click "Create Warp". Enter the URL of the page you want to extract data from.

Step 2: Write Your Prompt

Describe what data you want to extract in natural language. Be specific about the fields you need.

Step 3: Generate Schema

Click "Generate Schema". The AI will analyze your prompt and create an OpenAPI schema. This takes 10-30 seconds. Review the generated fields to ensure they match your needs.

Step 4: Deploy

Click "Deploy Warp" to deploy your warp. The first extraction starts immediately. You can watch the agent work in the Monitor tab.

Step 5: Access Your Data

Once extraction completes, your data is available via the Stargate API:

curl "https://{your-warp-id}.stargate.warpstack.ai/?limit=10"

That's it! Your data is now accessible as a REST API. Set a schedule to keep it fresh, or trigger manual runs when you need updates.

Usage & Billing

Warpstack offers custom enterprise pricing based on your data volume and requirements. Contact our sales team for a tailored quote.

Enterprise Plan

All enterprise agreements include full platform access, dedicated support, custom SLAs, and compliance features.

• Full audit trail: Chain of custody logging for every data point.
• PII detection & redaction: Automatic detection and redaction at the edge.
• Custom SLAs: Uptime guarantees and response time commitments.
• Dedicated support: Priority support with dedicated account manager.

Usage Components

Enterprise usage is charged based on three components. Rates are included in your custom enterprise agreement.

Extraction

Token-based usage during extraction. The rate is a blended measure combining input tokens (page content) and output tokens (extracted data).

• V1-Lite: For simple, straightforward pages
• V1-Base: For complex sites with dynamic content

Expect 20,000+ tokens per simple extraction. Complex pages use more.

Storage

Billed monthly based on total data stored across all warps.

• Included: Generous storage allocation in enterprise plans

Serving (Bandwidth)

Based on data transferred when your applications call the Stargate API.

• Included: API bandwidth in enterprise plans

How Enterprise Billing Works

• Custom agreements: Pricing tailored to your volume and requirements.
• Invoiced billing: Monthly invoicing with NET 30 terms.
• Usage tracking: Monitor your usage in the dashboard.
• Dedicated support: Contact your account manager for any billing inquiries.

Intelligence Tier Comparison

• V1-Lite: For simple, straightforward pages.
• V1-Base: Recommended. For complex sites with dynamic content.

Start with V1-Base for new warps. See Intelligence Tiers for details.

Account & Authentication

Creating Your Account

Creating a Warpstack account takes less than a minute. Once registered, you'll have immediate access to the platform and can start building your first data acquisition agent right away.

Getting Started

What Happens After Registration

Phone Verification

Phone verification is the first step of Warpstack's two-factor verification process. A 6-digit verification code is sent via SMS to confirm you have access to the mobile number you provided during registration.

How Phone Verification Works

Resending Verification Code

If you didn't receive the SMS or the code expired, you can request a new one:

Phone Number Format

Email Verification

Email verification is the second step of Warpstack's two-factor verification process, completed after phone verification. This confirms your email address for notifications, billing, and account recovery.

Why Verification is Required

Verification Process

Resending the Verification Email

If you didn't receive the verification email or the link expired, you can request a new one:

Logging In

Access your Warpstack account using your registered email address and password. Sessions remain active for 60 minutes of inactivity before requiring re-authentication.

How to Log In

Session Management

Password Management

Keep your account secure by using a strong password and changing it periodically. Warpstack provides secure password change and reset functionality.

Password Requirements

Changing Your Password

Change your password anytime from the Settings page while logged in:

Forgot Your Password?

If you've forgotten your password, you can reset it via email:

User Guide

Creating Your First Warp

A warp is an autonomous data extraction and serving job that acquires structured data from any website and serves it as a production REST API. The quality of your warp depends entirely on the quality of your schema, which depends on the quality of your prompt. This guide will walk you through each step in detail.

Step 1: Enter the Target URL

Start by entering the URL of the page you want to extract data from. This should be the specific page containing the data, not just the homepage.

• Be specific: Use example.com/products not just example.com
• Public pages: The agent cannot log in or bypass authentication
• Static content: Works best on pages where data loads with the page, not infinite scroll

Step 2: Write Your Prompt

The prompt is the most important input. It tells the AI what data structure to generate and instructs the browser agent on what to extract. Be as specific and detailed as possible.

What Makes a Good Prompt

A good prompt answers these questions:

• What data do you need? List every field explicitly (name, price, description, image URL)
• What type is each field? Specify if something should be a number, boolean, or date
• Where is the data located? Mention sections of the page (product grid, pricing table, sidebar)
• What are the edge cases? Describe variations (items with/without discounts, optional fields)
• What should be excluded? Mention what to skip (ads, related products, navigation)

Prompt Examples

Good prompt:

Bad prompt:

More Prompt Examples by Use Case

• SaaS Pricing:

  "Extract pricing plans from the /pricing page. For each plan: name, monthly price (number), yearly price (number), list of features as an array of strings, whether it's marked as 'Popular' or 'Recommended' (boolean), and any limits mentioned (e.g., '10 users', '50GB storage')."

• Job Listings:

  "Extract job postings from the careers page. Get: job title, department, location (city and country as separate fields), employment type (full-time/part-time/contract as an enum), salary range if shown (min and max as numbers), and the date posted in ISO format."

• News Articles:

  "Extract articles from the news homepage. For each: headline (string), summary/description (string), author name, publication date as ISO datetime, category/section, and the article URL. Limit to the main news grid, exclude sidebar widgets and advertisements."

Step 3: Generate the Schema

Click "Generate Schema" to create an OpenAPI schema from your prompt. This typically takes 10-30 seconds. You'll see the schema stream in real-time as the AI builds it.

What Happens During Generation

• Input Cleaning: Your prompt is refined to remove filler words and clarify intent
• Schema Generation: AI creates field names, types, descriptions, and validations
• Validation: The schema is validated against OpenAPI 3.1.0 and Structured Outputs rules
• Preview Data: Sample data is generated to show what your API will return

If Generation Fails

Schema generation can fail for several reasons:

• Validation errors: The schema violated Structured Outputs rules. Try simplifying your prompt.
• Ambiguous prompt: The AI couldn't determine what you wanted. Be more specific.
• Complex nesting: Too many nested objects. Flatten your data structure.
• Retry: Sometimes it's transient — click "Generate" again.

Step 4: Review the Generated Schema

Always review the generated schema before deploying. Check that:

• All fields you need are present: The AI may have missed something from your prompt
• Field types are correct: Prices should be numbers, not strings. Booleans for yes/no values.
• Field names make sense: They should be descriptive and consistent (snake_case recommended)
• Descriptions are accurate: The agent reads these to understand what to extract
• No unnecessary fields: Remove fields you won't use — they add cost

Review the Preview Data

The preview panel shows sample data based on your schema. This is AI-generated example data, not actual extracted data, but it helps you visualize what your API response will look like.

• Check that the data structure matches your expectations
• Verify nested arrays and objects look correct
• Ensure field types match what the site actually has

Step 5: Edit the Schema (Optional)

If the generated schema isn't quite right, you can edit it before deploying. You have two options:

Visual Mode (Recommended for Beginners)

A no-code interface where you can:

• Click on fields to rename them or change their type
• Add new fields using the "Add Field" button
• Delete fields you don't need
• Edit descriptions to give the agent better instructions
• Add validations like enums, patterns, or min/max values

Code Mode (Advanced Users)

Direct JSON editing with syntax highlighting. Use this when you need:

• Complex nested structures
• Advanced OpenAPI features ($ref, allOf, anyOf)
• Precise control over validations
• Bulk editing (copy/paste from existing schemas)

Edit with AI

Type natural language instructions to modify your schema:

• "Add a 'category' field with enum values: electronics, clothing, home"
• "Change 'price' to a number type with minimum 0"
• "Remove the 'description' field"
• "Add a nested 'seller' object with 'name' and 'rating' fields"

Step 6: Configure Settings

Before deploying, configure how your warp will run:

Intelligence Tier

Choose the AI model that powers your extraction agent:

• V1-Lite: For simple, straightforward pages.
• V1-Base (recommended): For complex sites with dynamic content.

See Intelligence Tiers for details.

Refresh Schedule

Set how often the agent should run and extract fresh data:

• Manual: Only runs when you trigger it. Best for testing or one-off extractions.
• Hourly: 24 runs per day. For real-time data like stock prices or inventory.
• Daily: Once per day. Most common choice for catalogs, listings, news.
• Weekly: Once per week. For slowly-changing data like documentation or reports.

Step 7: Deploy Your Warp

Click "Deploy Warp" to deploy your warp. Several things happen:

• Schema locks: Your schema becomes immutable to protect API consumers
• First extraction starts: The agent immediately begins extracting data
• API endpoint activates: Your unique Stargate URL becomes live
• Schedule begins: If you set a schedule, the warp will run automatically

After deployment, you'll be redirected to the Monitor tab where you can watch the agent work in real-time.

Your API Endpoint

Your warp gets a unique URL: https://{warp-id}.stargate.warpstack.ai. This is a read-only endpoint that returns your extracted data as JSON. Endpoints are public by default, but you can secure them with access tokens from the Monitor page.

Schema Editor Guide

The schema editor is where you refine your data structure. Your schema defines both the API contract (what consumers will receive) and the agent instructions (what to extract). Getting it right is critical.

Visual Mode

A no-code interface for editing schemas. Switch to Visual mode using the toggle at the top of the editor.

Adding a Field

Field Types

• String: Text data (names, descriptions, URLs)
• Number: Decimal numbers (prices, ratings, percentages)
• Integer: Whole numbers (counts, quantities, years)
• Boolean: True/false values (in_stock, is_featured, has_discount)
• Array: Lists of items (features, images, tags)
• Object: Nested structures (address with city/state/zip)

String Validations

• Enum: Fixed list of allowed values (e.g., "monthly", "yearly")
• Pattern: Regex validation (max 500 characters)
• Format: Predefined formats — date-time, date, time, email, hostname, ipv4, ipv6, uuid

Number Validations

• Minimum: Lowest allowed value (e.g., 0 for prices)
• Maximum: Highest allowed value (e.g., 5 for ratings)
• Multiple Of: Decimal precision (e.g., 0.01 for prices = $29.99)

Code Mode

Direct JSON editing for advanced users. The schema follows OpenAPI 3.1.0 specification.

Editor Features

• Syntax highlighting for JSON
• Real-time validation with error highlighting
• Line numbers and bracket matching
• Find and replace (Cmd/Ctrl + F)

Common Code Mode Tasks

• Adding $ref for reusable component definitions
• Using anyOf for union types (string or null)
• Setting complex regex patterns
• Bulk editing field descriptions

AI Schema Modification

Use natural language to edit your schema. Click "Edit with AI" and describe what you want to change:

• "Add a 'currency' field with enum: USD, EUR, GBP"
• "Change 'price' from string to number with 2 decimal precision"
• "Make 'discount_price' nullable (can be null if no discount)"
• "Add a nested 'dimensions' object with width, height, depth as numbers"
• "Remove all fields related to shipping"

Writing Good Field Descriptions

Field descriptions are critical — the extraction agent reads them to understand what to look for. A good description tells the agent exactly what to extract and where to find it.

Description Examples

• Good:

  "The current sale price displayed in the product card, as a decimal number without currency symbol (e.g., 29.99)"

• Bad:

  "Price"

• Good:

  "Boolean indicating if the product is currently in stock. True if 'In Stock' or 'Available', false if 'Out of Stock' or 'Sold Out'"

• Bad:

  "Stock status"

Understanding Schemas

Your schema is the foundation of your warp. It defines two things simultaneously: the structure of your API (what developers consume) and the instructions for the extraction agent (what to look for). A well-designed schema leads to accurate, consistent extractions.

Schema Structure

Warpstack schemas follow the OpenAPI 3.1.0 specification. Here's the basic structure:

• info.title: A descriptive name for your API
• info.description: High-level instructions for the agent about what to extract
• paths: Defines your API endpoint (always "/" for Warpstack)
• components.schemas: Your data models with field definitions

The info.description Field

This is crucial — the agent reads this to understand its overall mission. Include:

• What type of data you're extracting
• Where on the page to look
• What to include and exclude
• How to handle edge cases

Structured Outputs Rules

Warpstack uses OpenAI's Structured Outputs feature, which has strict schema requirements. Violating any of these rules will cause validation errors.

Rule 1: All Properties Must Be Required

Every property you define must be listed in the required array. Optional fields are not supported — if a field might not exist, use anyOf with null.

Rule 2: additionalProperties Must Be False

Every object (root and nested) must include "additionalProperties": false. This prevents the agent from adding unexpected fields.

Rule 3: Arrays Must Define Items

Every array type must have an items property defining the element type.

Rule 4: Forbidden Keywords

Do not use these OpenAPI keywords — they're not supported by Structured Outputs:

• example — use description instead
• default — not supported
• minLength / maxLength — use pattern instead
• format: "uri" — use "hostname" or no format

Allowed String Formats

These are the only valid format values for strings:

• date-time — ISO 8601 datetime (2025-01-15T10:30:00Z)
• date — ISO 8601 date (2025-01-15)
• time — ISO 8601 time (10:30:00)
• duration — ISO 8601 duration (PT1H30M)
• email — Email address
• hostname — Domain name
• ipv4 — IPv4 address
• ipv6 — IPv6 address
• uuid — UUID format

Schema Limits

• Nesting: Maximum 5 levels deep (prefer 3 or fewer)
• Properties per object: Maximum 100 (prefer under 50)
• Total fields: Maximum 500
• Regex patterns: Maximum 500 characters

Monitoring Your Warps

After deploying a warp, you can watch the extraction agent work in real-time. The Monitor tab provides full observability into what your agent is doing.

Live Agent View

See exactly what your agent sees. The browser view shows real-time screenshots as the agent navigates the target site:

• Browser screenshots: Updated every few seconds during extraction
• Highlighted actions: See what the agent is clicking or reading
• Navigation steps: Watch the agent move through pages

Activity Log

The activity log shows the agent's reasoning and actions as it extracts data:

• Step-by-step actions: "Navigating to URL", "Extracting products", "Found 24 items"
• Agent reasoning: Why it made certain decisions
• Timestamps: How long each step took
• Errors: What went wrong if extraction fails

Execution Status

Key metrics displayed during and after extraction:

• Status: Running, Completed, or Failed
• Records: Number of items extracted
• Duration: Total extraction time
• Cost: Tokens consumed × price per token

Manage Access Token

From the Monitor page, you can secure your API endpoint with an access token. Click the link icon next to the API URL at the top of the page to open the access token manager.

• No token configured: Your endpoint is publicly accessible (default)
• Generate Access Token: Creates a secure token to require authentication
• Copy token: Token is shown once—copy it immediately
• Delete token: Revokes access and makes endpoint public again

See Access Tokens in the Stargate API section for complete usage details.

Understanding Token Usage

Token usage during extraction depends on several factors:

• Page complexity: More content = more tokens to process
• Schema size: More fields = more output tokens
• Navigation depth: Multi-page extractions use more tokens
• Visual analysis: Processing screenshots adds ~20-30% more tokens

Expect 20,000+ tokens for simple extractions. Complex sites with lots of content or multi-step navigation will use significantly more.

Managing Deployed Warps

Once a warp is deployed, the schema is locked to protect API consumers from breaking changes. You can still make changes, but you need to follow a specific process.

Schema Locking

When you deploy a warp, its schema becomes locked. This means:

• Your API continues working: While you edit, consumers keep getting the old schema
• No accidental changes: You can't accidentally break your API
• Explicit deployment: Changes only go live when you click "Deploy Warp"

Editing an Active Warp

To modify a deployed warp's schema:

Breaking Changes to Avoid

Some changes will break existing API consumers. Avoid these unless you're coordinating with all consumers:

• Renaming fields: Old field names will no longer exist
• Changing field types: String to number will break JSON parsing
• Removing fields: Consumers expecting them will error
• Restructuring: Moving fields or changing nesting

Manual Runs

You can trigger an extraction manually at any time, regardless of schedule. Click "Run Now" in the warp header. Use this for:

• Testing after schema changes
• Getting fresh data immediately
• Debugging extraction issues
• One-off data collection (when schedule is set to Manual)

Pausing a Warp

Pausing stops scheduled runs but keeps everything else active:

• Your API endpoint remains available
• Existing data is still served
• No new extractions occur (no new costs)
• Resume anytime to restart the schedule

Deleting a Warp

Deleting a warp is permanent:

• Your API endpoint stops working immediately
• Data is retained for 30 days, then permanently deleted
• This action cannot be undone

Scheduling

Scheduling controls how often your warp runs and extracts fresh data. Choose based on how frequently the source data changes and your cost tolerance.

Available Schedules

Manual

No automatic runs. The warp only extracts data when you click "Run Now". Best for:

• Testing and development
• One-off data collection projects
• Infrequently updated sources

Hourly

Runs every hour, 24 times per day. Best for:

• Real-time price monitoring
• Inventory tracking
• Breaking news feeds
• Stock data (during market hours)

Daily

Runs once per day at a consistent time. The most common choice, balancing freshness and cost. Best for:

• Product catalogs
• Pricing pages
• Job listings
• News aggregation

Weekly

Runs once per week. Best for:

• Market research reports
• Documentation pages
• Reference data
• Slowly changing content

Schedule Cost Impact

Schedule frequency is the biggest cost driver. Consider carefully:

• Hourly: 720 runs/month — ~30x the cost of daily
• Daily: 30 runs/month — baseline cost
• Weekly: ~4 runs/month — ~1/7 the cost of daily

Always choose the lowest frequency that meets your data freshness requirements.

Intelligence Tiers

Intelligence tiers determine which AI model powers your extraction agent. Different models have different capabilities and costs. The "per 1M tokens" price is a blended rate combining both input tokens (page content) and output tokens (extracted data).

Both tiers use vision AI to see and understand web pages like a human. The difference is in capability and cost.

V1-Lite

Fast and cost-efficient. Best for straightforward pages with clear structure.

Best For

• Product listings and directories
• Blog posts and articles
• Simple contact pages
• High-volume extraction where cost matters

V1-Base — Recommended

More powerful AI for complex sites. Handles dynamic content and tricky interfaces.

Best For

• Dynamic JavaScript rendering
• Infinite scroll and pagination
• JavaScript-heavy applications
• Complex layouts and navigation
• Charts, dashboards, and visual data

Choosing the Right Tier

Start with V1-Base for most use cases. It's only 2x the cost of V1-Lite but handles a much wider range of sites. Only use V1-Lite if:

• The target site has straightforward, simple pages
• You've tested and V1-Lite provides acceptable accuracy
• You're running high-volume extractions and need to minimize cost

Stargate Data API

API Overview

The Stargate API provides high-performance, read-only access to your extracted data with SQL-like querying capabilities.

Endpoints

GET https://{warp-id}.stargate.warpstack.ai/

{
  "data": [...],              // Your data array
  "total": 156,               // Total record count
  "available_columns": [...], // All available column names
  "count": 25                 // Records in this response
}

GET https://{warp-id}.stargate.warpstack.ai/stats

{
  "total_records": 156,
  "storage_bytes": 524288,
  "total_sessions": 12,
  "total_sessions_duration_ms": 540000
}

Access Tokens

By default, Stargate endpoints are publicly accessible. You can secure your endpoint by generating an access token, which requires all API requests to include authentication.

Generating an Access Token

Using Access Tokens

Once a token is configured, all requests to your endpoint must include authentication. There are two methods:

curl https://{warp-id}.stargate.warpstack.ai \
  -H "Authorization: Bearer {your-token}"

GET https://{warp-id}.stargate.warpstack.ai?access_token={your-token}

Revoking Access

To revoke an access token, open the access token manager and click the delete button. Once deleted, the endpoint becomes publicly accessible again.

Column Selection

Use the select parameter to specify which columns to return. If omitted, returns the first 50 columns.

?select=id,name,email

?select=user.name,user.email,address.city

?select=items[].name,items[].price

Filtering

Filter data using filter=column:operator:value syntax. Combine multiple filters with commas (AND logic).

?filter=status:eq:active,price:lt:100

Available Operators

Filter Examples

Sorting

Sort results using sort=column:direction. Direction is asc (ascending) or desc (descending).

Pagination

Control how many records to return. Default limit is 100, maximum is 10,000.

?limit=25&offset=50

?limit=25&cursor=eyJpZCI6MTIzfQ

{
  "data": [...],
  "cursor": "eyJpZCI6MTUwfQ",  // Use this for next request
  "total": 500
}

Output Formats

?output=json

?output=csv

Usage Examples

cURL

curl "https://my-warp.stargate.warpstack.ai/?limit=10"

curl "https://my-warp.stargate.warpstack.ai/?filter=price:lt:100&sort=price:asc&limit=20"

curl "https://my-warp.stargate.warpstack.ai/?output=csv" > data.csv

JavaScript / Fetch API

const response = await fetch(
  'https://my-warp.stargate.warpstack.ai/?limit=10&sort=created_at:desc'
);
const { data, total } = await response.json();

console.log(`Retrieved ${data.length} of ${total} total records`);
data.forEach(item => console.log(item));

Python

import requests

# Basic query
response = requests.get(
    'https://my-warp.stargate.warpstack.ai/',
    params={
        'limit': 10,
        'sort': 'created_at:desc',
        'filter': 'status:eq:active'
    }
)
data = response.json()

print(f"Retrieved {len(data['data'])} records")
for item in data['data']:
    print(item)

Combining Parameters

You can combine multiple query features for powerful data access:

?select=name,price,rating&filter=price:lt:100,rating:gte:4&sort=price:asc&limit=20&offset=0

Returns name, price, and rating for items under $100 with rating ≥4, sorted by price ascending, first 20 results.

Error Handling

Workbench

Data Viewer

The Workbench is your data exploration tool. View, filter, sort, and export your extracted data.

Accessing Workbench

Click on any warp in your dashboard, then navigate to the "Workbench" tab to view your data in table format.

Filtering & Sorting Data

Exporting Data

curl "https://my-warp.stargate.warpstack.ai/?output=csv&limit=10000" > export.csv

Metrics & Costs

Understanding Metrics

Warpstack provides detailed metrics to help you understand your usage and costs. The dashboard displays real-time data on extraction usage, storage consumption, and API serving across customizable time ranges.

Viewing Your Metrics

Access your metrics from the Usage section in the dashboard. You can view usage across different time periods:

• Day: Last 24 hours of activity
• Week: Rolling 7-day window
• Month: Current billing cycle (30 days)
• Total: All-time cumulative usage

Key Metrics Tracked

Warpstack tracks three categories of usage that contribute to your costs:

• Extraction (Tokens): LLM tokens consumed during data extraction runs
• Storage (GB): Total data stored across all your warps
• Serving (Bandwidth): Data transferred via Stargate API calls

Cost Breakdown

All Warpstack costs are usage-based. You only pay for what you use, with no hidden fees or minimums.

Extraction Costs (Token-Based Pricing)

Extraction costs are based on LLM token consumption. The "per 1M tokens" price is a blended rate that combines both input tokens (the page content sent to the model) and output tokens (the extracted data returned). This single metric simplifies cost calculation.

Intelligence Tiers

• V1-Lite: For simple, straightforward pages.
• V1-Base: Recommended. For complex sites with dynamic content.

Extraction Usage Example

A simple product extraction using V1-Base:

• ~25,000 tokens per extraction
• Token usage scales with page complexity and schema size
• Request a demo to discuss pricing based on your volume requirements

Storage

Storage is billed monthly based on the total data stored across all your warps. Extracted data is stored as optimized JSON and is typically very space-efficient.

• 100,000 records: ~100 MB
• 1 million records: ~1 GB
• Included: Generous storage allocation in enterprise plans

Serving (API Bandwidth)

Serving is based on the bandwidth consumed when your applications fetch data via the Stargate API. This includes all GET requests to your warp endpoints.

• Typical response size: 1-50 KB depending on data volume
• Included: API bandwidth in enterprise plans

Cost Optimization

Optimize your Warpstack costs by following these strategies to reduce token usage and improve efficiency.

Choose the Right Intelligence Tier

Start with V1-Base for most sites. If the target is simple and you need to minimize cost, test V1-Lite. See Intelligence Tiers for details.

Optimize Refresh Frequency

The biggest cost lever is how often your warp runs. Choose the minimum frequency that meets your needs:

• Hourly: Real-time monitoring, price tracking (highest cost)
• Daily: Most common for catalog data, news, listings
• Weekly: Slowly changing data, documentation, reference data
• Manual: One-time extractions, on-demand updates (lowest cost)

Switching from hourly to daily reduces costs by ~24x. Daily to weekly reduces by another ~7x.

Write Targeted Prompts

Specific, focused prompts reduce token usage by helping the agent work more efficiently:

• Include the exact URL path if known (e.g., "Extract from /pricing")
• Specify exactly what data you need — avoid "get everything"
• Mention where on the page the data appears (header, table, footer)
• Describe edge cases upfront to avoid retries

Simplify Your Schema

Every field in your schema adds to extraction complexity and token usage. Only extract what you'll actually use:

• Remove fields you don't need in your application
• Avoid deeply nested structures when flat data works
• Use simple types (string, number) over complex validations when possible

Monitor and Pause Unused Warps

Regularly review your active warps. Pause or delete warps that are no longer needed to stop accumulating extraction costs. You can always re-deploy a paused warp later.

Understanding Your Balance

The Usage section shows your current balance and projected runway:

• Month-to-Date: Total spent so far this billing cycle
• Burn Rate: Average daily spending based on recent usage
• Runway: Days until balance depletes (Balance ÷ Daily Burn Rate)

Activity & Notifications

Activity Feed

Activity Feed

Track all warp activity in one timeline. See what's happening across your account.

Notification Center

Stay informed about important events. Access via the bell icon in the top navigation.

Settings

Profile Settings

Customize your account, security, notifications, and preferences.

Profile Settings

Manage your account information and contact details.

Security Settings

Change your password from the Settings page:

Notification Preferences

Control which email notifications you receive.

User Preferences

Troubleshooting

Schema Validation Errors

These are the most common schema errors and how to fix them.

{
  "properties": {
    "name": {...},
    "email": {...}
  },
  "required": ["name"]
}

{
  "properties": {
    "name": {...},
    "email": {...}
  },
  "required": ["name", "email"]
}

{
  "tags": {
    "type": "array",
    "description": "Tags"
  }
}

{
  "tags": {
    "type": "array",
    "description": "Tags",
    "items": {"type": "string"}
  }
}

{
  "type": "object",
  "required": ["name"],
  "properties": {
    "name": {"type": "string"}
  },
  "additionalProperties": false  // ✅ Add this
}

Deployment Failures

Extraction Failures

Getting Help

Best Practices & Examples

Writing Effective Prompts

Your prompt is the foundation of every warp. A well-written prompt produces a better schema, which produces better data. Spend an extra minute here to save hours of debugging later.

Schema Design Tips

Use Case Examples

Reference

Glossary

Warp

An autonomous data extraction and serving job. Configured with a prompt and optional target URL, runs on a schedule you define, and serves data via a production API.

Stargate

The high-performance API layer that serves your extracted data via unique URLs:https://{warp-id}.stargate.warpstack.ai

Schema

The data contract defining your API structure. Written in OpenAPI 3.1.0 format with Structured Outputs compliance.

Structured Outputs

OpenAI's strict JSON schema format requiring all fields to be required, no additional properties, and specific validation rules. Ensures reliable, structured AI responses.

Intelligence Tier

The AI model used for extraction. V1-Lite for simple pages, V1-Base for complex content.

Visual Analysis

Enables vision AI capabilities allowing the agent to "see" the page like a human. Critical for charts, dynamic dashboards, and visual content.

Extraction

The process of gathering data from web pages using autonomous browser agents.

Deployment

Making a warp active/live. Locks the schema and begins data extraction. Creates your Stargate API endpoint.

Schema Locking

Protection mechanism that prevents accidental schema changes to deployed warps. Ensures API stability for consumers.

Account Balance

Prepaid funds used for usage-based billing. Add funds to your balance, usage costs are deducted in real-time.

Runway

Days until your balance runs out, calculated from current balance and daily burn rate. Helps plan when to add funds.

Burn Rate

Average daily spending based on recent usage patterns. Used to calculate runway.

Tokens

Units of AI model consumption. Roughly 4 characters = 1 token. Used to measure and bill extraction costs.

Frequently Asked Questions