Methodology · Overview
Usage Modeling Methodology
Reviewed by Byron Malone · Last reviewed .
Primary sources
Tokenization details are documented by each provider: OpenAI's tiktoken library (github.com/openai/tiktoken) with BPE tokenization for GPT models, and Anthropic's tokenizer documentation. As a practical approximation: English text tokenizes at approximately 3.8-4.2 characters per token (we use 4.0 as the default). Code tokenizes at approximately 3.0 characters per token (more tokens per character due to special tokens for syntax).
Provider API quotas and rate limits are documented in each provider's API reference. Anthropic: tokens-per-minute (TPM) limits vary by tier from 50,000 TPM (Tier 1) to 4,000,000 TPM (Tier 4). OpenAI: similar tier structure. These limits affect cost at high volumes (you may need to upgrade your tier).
Monthly cost projection model
Monthly cost = daily_requests × avg_input_tokens × input_rate + daily_requests × avg_output_tokens × output_rate, × 30. We parameterize by: (1) requests per day (from user input or estimated from DAU × requests_per_user); (2) average input tokens (estimated from use case description × 4 chars/token heuristic); (3) average output tokens (estimated from response length target).
Input token estimation by use case: customer support chat (80-200 input tokens per turn, 50-150 output), document summarization (500-4,000 input, 100-500 output), RAG-based Q&A (500-2,000 input including context chunks, 100-300 output), code generation (200-1,000 input, 200-800 output).
Cost scaling
LLM cost scales linearly with volume (no volume discount at API list prices). We model breakeven points for provider tier upgrades and cached-vs-uncached strategies. At >10M tokens/month, enterprise pricing negotiations typically begin and can reduce costs 20-50% below list.
Limitations
Token counts are estimates until you instrument your actual application. The 4-char/token heuristic fails for non-English languages (Japanese and Chinese have higher token efficiency; some languages have lower), code with many special characters, and heavily formatted text (JSON, XML). Use provider tokenizer libraries for production cost planning.
Update protocol
This category is reviewed quarterly. Immediate updates are triggered by changes to the primary source documents listed in the citations above — rate table revisions, new agency guidance, or regulatory amendments.
Error reports go to info@bedrockatools.com. Corrections are published on our corrections page.