Inference Time Estimator

Estimate AI model inference time and batch processing performance

Request Parameters

AI Model

~0.5s base latency • Max concurrency: 10

Prompt Tokens(e.g., 200)

Completion Tokens(e.g., 100)

Concurrency Level(max: 10)

Batch Size(e.g., 100)

Total Tokens

300

Effective Concurrency

0.65s

Base: 0.5s + Token processing: 0.15s

13.0 seconds

100 requests ÷ 5 concurrent

7.7 req/s

462 requests per minute

Performance Notes

• Higher concurrency may increase latency
• Token processing time varies by model
• Network latency not included in estimates

GPT-3.5 Turbo

0.65s per request

7.7 req/s throughput

GPT-4

2.38s per request

2.1 req/s throughput

GPT-4 Turbo

1.75s per request

2.9 req/s throughput

Claude 3 Haiku

1.00s per request

5.0 req/s throughput

Claude 3 Sonnet

1.50s per request

3.3 req/s throughput

Claude 3 Opus

3.50s per request

0.9 req/s throughput