Amniscient Detection SDK for TypeScript

TypeScript/JavaScript client for the Amniscient Inference API over HTTPS or NATS (WebSocket): low-latency object detection behind a single call, client.detect(image, modelId).

Requires Node 22+ (native fetch and WebSocket) or a modern browser. One runtime dependency: @nats-io/nats-core, for the NATS transport.

Installation

npm install @amniscient/detection

Quick start

Option A — the snippet. Drop this into your app:

import { DetectionClient } from "@amniscient/detection";
const client = new DetectionClient({
  endpoint: "https://api.amniscient.com/inference",
  apiKey: "YOUR_KEY",
  organizationId: "YOUR_ORG",
});
const result = await client.detect("photo.jpg", "MODEL_ID");
console.log(result.detectionFound);                 // true / false
for (const d of result.detections) {
  console.log(d.classId, d.confidence.toFixed(2), d.bbox); // bbox: {x1,y1,x2,y2}, normalized 0-1
}

Option B — a runnable script. Save this as detect.ts, fill in the four values at the top, and run it with npx tsx detect.ts path/to/image.jpg:

import { DetectionClient } from "@amniscient/detection";
const ENDPOINT = "https://api.amniscient.com/inference";
const API_KEY = "YOUR_KEY";
const ORG_ID = "YOUR_ORG";
const MODEL_ID = "MODEL_ID";
const imagePath = process.argv[2];
if (!imagePath) {
  console.error("Usage: npx tsx detect.ts <image_path>");
  process.exit(1);
}
const client = new DetectionClient({ endpoint: ENDPOINT, apiKey: API_KEY, organizationId: ORG_ID });
const result = await client.detect(imagePath, MODEL_ID);
console.log(`Found ${result.detections.length} objects in ${imagePath}:\n`);
for (const d of result.detections.sort((a, b) => b.confidence - a.confidence)) {
  const b = d.bbox;
  console.log(`  ${(d.confidence * 100).toFixed(0)}%  ${d.classId}`);
  console.log(`     bbox (0-1): (${b.x1.toFixed(3)}, ${b.y1.toFixed(3)}) -> (${b.x2.toFixed(3)}, ${b.y2.toFixed(3)})`);
}

Top-level await needs "type": "module" in your package.json (or name the file detect.mts). A ready-to-run version that reads these from env vars lives in examples/basic-detection.ts.

detect() accepts a file path (Node), raw bytes (Uint8Array/ArrayBuffer), or a Blob (browser). Everything is async — the client is safe to construct once and reuse.

Endpoint. Use the HTTPS endpoint https://api.amniscient.com/inference. Your apiKey is your own application API key — validated by the API gateway on every request. (This SDK is HTTP-only. For NATS, use the Python SDK, amniscient-detection 0.5.0 or later.)

How it works

The whole surface is detect(). You give it an image and a model; it returns a DetectionResult. Everything in between is handled for you:

1. Auth is validated first — a missing API key throws AuthenticationError before any image work or network call happens. It is never skipped.
2. The image is prepared — loaded into bytes, optionally compressed, and validated.
3. The request is sent over HTTPS or NATS and, on a transient failure (network error, timeout or 5xx), retried with exponential backoff.
4. The reply becomes a DetectionResult, or a specific typed error is thrown (see Error Handling).

For contributors, the pieces are small and single-purpose: client.ts orchestrates the flow and owns retries + error mapping; transport.ts moves bytes over HTTP or NATS and nothing else; contract.ts is generated from sdk/contract/contract.json; types.ts holds the public shapes plus the filterBy* helpers; utils.ts is where a reply becomes a DetectionResult and where image bytes are loaded/validated; annotate.ts turns a payload into draw-ready boxes; errors.ts holds the typed error hierarchy and the HTTP-status mapping.

Features

• One call, detect() — connection handling, retries, and parsing done for you
• HTTPS or NATS — the same calls over either; NATS keeps one WebSocket open for lower latency
• Built on native fetch/FormData/Blob/WebSocket — Node 22+ or a modern browser
• Pluggable compression — passthrough by default; supply a compress hook (e.g. backed by sharp or a <canvas>) to actually resize/re-encode
• Retry logic — exponential backoff for transient errors (network, 5xx)
• Validated auth — a missing/invalid API key throws AuthenticationError, never a silent failure
• Fully async — every method returns a Promise; fan out with Promise.all when you need concurrency
• Raw mode — detectRaw() returns the unparsed server payload when you need it
• Full type hints — .d.ts shipped for every export, IDE-friendly

Usage Examples

Different image inputs

import { readFile } from "node:fs/promises";
import { DetectionClient } from "@amniscient/detection";
const client = new DetectionClient({ endpoint: ENDPOINT, apiKey: API_KEY, organizationId: ORG_ID });
// From a file path (Node only)
const a = await client.detect("image.jpg", "MODEL_ID");
// From raw bytes (Uint8Array)
const bytes = new Uint8Array(await readFile("image.jpg"));
const b = await client.detect(bytes, "MODEL_ID");
// From an ArrayBuffer
const c = await client.detect(bytes.buffer, "MODEL_ID");
// From a Blob (browser, or Node 18+)
const blob = new Blob([bytes], { type: "image/jpeg" });
const d = await client.detect(blob, "MODEL_ID");

Filtering results

import { filterByConfidence, filterByClassId, filterByTag } from "@amniscient/detection";
const result = await client.detect("image.jpg", "MODEL_ID");
// Filter by confidence threshold
const highConf = filterByConfidence(result, 0.8);
// Filter by class id
const peopleOnly = filterByClassId(result, "person", "pedestrian");
// Filter by a tag key/value
const shelfA1 = filterByTag(result, "shelf", "A1");

Each filterBy* helper returns a new DetectionResult — the original is left untouched.

Raw responses

By default detect() parses the server reply into a DetectionResult. Use detectRaw() when you need the unparsed response instead — for logging/debugging, or when downstream code already depends on the original xywh contract:

const payload = await client.detectRaw("image.jpg", "MODEL_ID");
// payload == { inference_id: "...", detection_found: true,
//              xywh: [{ x, y, width, height, confidence, id, tag }, ...] }
for (const det of payload.xywh as Array<Record<string, unknown>>) {
  console.log(det.id, det.confidence);
}

Retries, error mapping, and auth validation are unchanged — only the return value skips parsing.

Barcode & QR reading

Pass detectBarcodes: true to decode barcodes/QR codes alongside object detection, in the same request:

const result = await client.detect("shelf.jpg", "MODEL_ID", { detectBarcodes: true });
// object detection is unchanged; barcodes ride alongside:
for (const code of result.barcodes) {
  console.log(code.type, code.value); // e.g. qr https://…  |  upc-a 0012345678905
}

It’s off by default — pass detectBarcodes: true explicitly. Supported symbologies (wire type values): qr, upc-a, upc-e, code-39.

Notes:

• result.barcodes is [] when detectBarcodes is off, no codes are found, or the decode step fails or times out — it never throws and never affects result.detections.
• Numeric 1D codes (UPC-A/UPC-E) decode to their expanded digit string (e.g. "0012345678905"), so don’t expect an exact round-trip of a shorter printed value. code-39 returns the encoded text as-is.

Pinning a model version

Models are versioned. By default a request runs whichever version is currently active, so results follow promotions automatically:

// active version — follows promotions
const result = await client.detect("shelf.jpg", "MODEL_ID");
// pinned — same weights and thresholds regardless of later promotions
const pinned = await client.detect("shelf.jpg", "MODEL_ID", { version: 3 });

Versions are 1-based. Pin when you need reproducible output — a benchmark, a regression suite, or a customer held to a specific model revision. Leave it unset to get improvements as they’re promoted. The version also selects that version’s confidence/IoU thresholds and detection-item tags, not just its weights. A version that doesn’t exist throws ModelNotFoundError.

Multi-tenant / per-request credentials

The client can hold one credential for its lifetime, or none at all. If you’re a multi-tenant proxy whose credentials arrive with each request and are never stored server-side, build one shared client with no stored credentials and pass apiKey / organizationId per detect() call — they ride each request, and nothing is kept between calls:

// one shared, long-lived client — no credentials stored
const client = new DetectionClient({ endpoint: ENDPOINT });
// each request carries its own tenant's credentials
const result = await client.detect(image, "MODEL_ID", {
  apiKey: tenantKey,
  organizationId: tenantOrg,
});

Over NATS (WebSocket)

For lower latency and a persistent connection, use the NATS transport. Pass transport: "nats" and the NATS endpoint; the same API key works. The organization is resolved from the key when you don’t pass it, and the NATS server confines the connection to that organization.

const client = new DetectionClient({
  endpoint: "wss://nats-server.amniscient.com",
  apiKey: "YOUR_KEY",
  transport: "nats",
});
const result = await client.detect("photo.jpg", "MODEL_ID");
const models = await client.listModels();
await client.close(); // drains the connection

• One tenant per client. NATS binds the key to the connection, so per-call apiKey / organizationId overrides throw a TypeError. Use one client per tenant, or HTTP for the multi-tenant proxy pattern.
• Same errors and retries as HTTP. A rejected key is an AuthenticationError; timeouts and 5xx replies are retried like HTTP.
• Call close() when you’re done, or the open WebSocket keeps Node running.

Validating credentials (login gate)

To accept an API key + org before a model is chosen (e.g. a web login step), use validate() — it needs no model id and no image:

if (!(await client.validate(organizationId, apiKey))) {
  throw new Error("Invalid API key or organization");
}

validate() resolves true/false — it calls GET /models, so an accepted key gets a 200 (a real positive signal) and a rejected key (401/403) → false. A missing route throws EndpointNotAvailableError rather than masquerading as valid, so you never get a false “looks valid” from a route that simply isn’t there — inspect statusCode on the thrown DetectionError if you need to tell those apart. Both validate() and getDefaultModel() are HTTP only and authenticate through the API gateway.

Listing models (model picker)

To let a user pick a model instead of pasting a UUID, listModels() returns the org’s selectable models — no model id or image needed:

const models = await client.listModels(organizationId, apiKey);
for (const m of models) {
  dropdown.add({ value: m.id, label: m.name }); // gray out non-TRAINED
  // each m: { id: "<uuid>", name: "<label>", status: "TRAINED" }
}
const dflt = await client.getDefaultModel(organizationId, apiKey); // model dict, or null

It returns top-level models (specialist sub-models are excluded), each with id, name, and status. HTTP only, authenticated through the gateway; organizationId / apiKey override the client’s defaults.

Putting it together — a typical flow

The methods above compose into one flow: validate the key at login → list models to pick one → detect on each capture. How you hold the client depends on your architecture.

Single-tenant app (one org, credentials known at startup) — one client for its lifetime:

const client = new DetectionClient({ endpoint: ENDPOINT, apiKey, organizationId });
// 1. Login gate — reject bad credentials before showing the app
if (!(await client.validate())) {
  throw new Error("Invalid API key or organization");
}
// 2. Let the user choose a model (dropdown), instead of pasting a UUID
const models = await client.listModels();      // [{ id, name, status }, ...]
const modelId = chooseFrom(models);            // e.g. first TRAINED model
// 3. Run detection on each capture — Promise.all when you need concurrency
const results = await Promise.all(captures.map((frame) => client.detect(frame, modelId)));
results.forEach((result) => handle(result.detections));

Multi-tenant proxy (credentials arrive per request, nothing stored) — one shared client, credentials passed per call:

import type { ImageInput } from "@amniscient/detection";
const client = new DetectionClient({ endpoint: ENDPOINT }); // no stored credentials
async function handleRequest(req: { org: string; key: string; image: ImageInput; modelId: string }) {
  // validate this tenant's key at their login step...
  if (!(await client.validate(req.org, req.key))) {
    return http401();
  }
  // ...then serve detections, credentials riding each call
  return client.detect(req.image, req.modelId, {
    organizationId: req.org,
    apiKey: req.key,
  });
}

Distinguish a real credential failure from a gateway routing error at the login gate — see Error Handling for the 401 vs 403 pattern.

Annotating detections (draw boxes)

Turn a /detect response + the source image into draw-ready boxes. Pass the raw payload (detectRaw()), unchanged:

const payload = await client.detectRaw("shelf.jpg", "MODEL_ID");
const result = await client.annotateDetections("shelf.jpg", payload); // label by name (default)
for (const b of result.boxes) {          // draw-ready specs — render however you like
  console.log(b.label, b.box);           // box = {x1,y1,x2,y2} in pixels; also id/name/confidence/color
}

result.boxes (an AnnotationBox[]) is the primary product — pixel-space, label-resolved specs you can render in any toolkit (sharp, canvas, SVG). Options: { label?: "name" | "id", color?: string, render?: boolean, names?: Record<string, string> } (label defaults to "name", color defaults to #00FF00).

You can also build the same boxes directly from a payload + known image dimensions, without a client, via the free function:

import { buildAnnotation } from "@amniscient/detection";
const boxes = buildAnnotation(payload, imgWidth, imgHeight, { label: "id" });

result.toPng() renders those boxes onto the source image and resolves with PNG bytes — but it needs a canvas backend (a browser OffscreenCanvas/<canvas>, or createImageBitmap). In a plain Node process it rejects with a message telling you to render .boxes yourself:

// Browser (or any environment with a canvas backend):
const png = await result.toPng();   // ready-to-display PNG bytes
// Plain Node: no canvas backend — render `.boxes` yourself, e.g. with
// sharp or @napi-rs/canvas, using the pixel-space box/label/color fields.

Handle the no-detections case — an empty payload throws NoDetectionsError instead of returning a blank image:

import { NoDetectionsError } from "@amniscient/detection";
try {
  const result = await client.annotateDetections(image, payload);
} catch (e) {
  if (e instanceof NoDetectionsError) {
    // nothing detected — show the original image / a "no detections" state
  } else {
    throw e;
  }
}

Name labels are pulled by id from the model’s detection-items (the metadata endpoint, a sibling of your inference endpoint), cached per model and falling back to the id. The SDK derives that URL from an https://…/inference endpoint; if yours differs, pass metadataEndpoint when constructing the client — or pass a names: { id: name } map to annotateDetections to skip the network call.

Configuration Options

ClientOptions passed to new DetectionClient({ ... }):

The compression truth: the base SDK ships no image library (zero runtime dependencies), so compression is a passthrough by default — bytes are sent unchanged even with compression: true. This matches the Python SDK’s behavior on a base install without Pillow. Supply compress (an ImageCompressor) to actually resize/re-encode; the jpegQuality / maxDimension / highCompression knobs are then passed to it as CompressOptions.

Error Handling

Every failure surfaces as a typed subclass of DetectionError, each carrying statusCode / responseBody (both undefined for client-side errors, i.e. ones that never reached an HTTP response):

import {
  AuthenticationError,
  ModelNotFoundError,
  ImageTooLargeError,
  RateLimitError,
  ServerError,
  ConnectionError,
} from "@amniscient/detection";
try {
  const result = await client.detect("image.jpg", "MODEL_ID");
} catch (e) {
  if (e instanceof AuthenticationError) console.error("Invalid API key or organization ID");
  else if (e instanceof ModelNotFoundError) console.error("Model not found:", e.message);
  else if (e instanceof ImageTooLargeError) console.error("Image too large:", e.message);
  else if (e instanceof RateLimitError) console.error("Rate limited — back off and retry");
  else if (e instanceof ServerError) console.error(`Server error (${e.statusCode})`);
  else if (e instanceof ConnectionError) console.error("Failed to connect to server");
  else throw e;
}

Transient failures (network errors, 5xx) are retried with exponential backoff before surfacing as ConnectionError or ServerError.

A missing route is not a rejected key. An API Gateway returns 403 MISSING_AUTHENTICATION_TOKEN for an unmatched route even when the key is valid. The SDK recognizes that signature and throws EndpointNotAvailableError (a distinct type — not AuthenticationError), so a route that simply isn’t deployed never masquerades as “bad credentials”:

import { EndpointNotAvailableError, AuthenticationError } from "@amniscient/detection";
try {
  const models = await client.listModels(organizationId, apiKey); // or detect(...)
} catch (e) {
  if (e instanceof EndpointNotAvailableError) {
    // the key is fine — the route just isn't exposed on this gateway/stage
    throw new Error("Inference route not available for this environment");
  } else if (e instanceof AuthenticationError) {
    throw new Error("Invalid API key or organization"); // genuine 401/403
  } else {
    throw e;
  }
}

validate() resolves true/false for a genuine accept/reject, and throws EndpointNotAvailableError when the route isn’t deployed — it never answers a confident false for a route that isn’t there.

Development

npm install
npm run build      # tsup -> dist (CJS + ESM + .d.ts)
npm test           # vitest run — 79 tests across 7 files
npm run typecheck  # tsc --noEmit

Run the example against a live env:

export AMNISCIENT_ENDPOINT="https://api.amniscient.com/inference"
export AMNISCIENT_API_KEY="..."; export AMNISCIENT_ORG_ID="..."; export AMNISCIENT_MODEL_ID="..."
npm run example -- path/to/image.jpg

Live sandbox test — proves a real detect() succeeds and a bad key is actually rejected, end to end against the sandbox environment:

bash scripts/run-sandbox-detect.sh [image]

(Requires an AWS SSO refresh first; re-run it if you hit an auth error.)

More detail lives alongside this package: docs/USER_GUIDE.md (task-oriented walkthrough), MIGRATION.md (moving off direct HTTP), BUILD.md (contributors), and tests/README.md (what each test file covers).

License

MIT