> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.amniscient.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.amniscient.com/_mcp/server.

# Amniscient Detection SDK for TypeScript

#### [TypeScript SDK](https://www.npmjs.com/package/@amniscient/detection)

NPM Repository

TypeScript/JavaScript client for the Amniscient Inference API over HTTPS or
NATS (WebSocket): low-latency object detection behind a single call,
`client.detect(image, modelId)`.

Requires Node 22+ (native `fetch` and `WebSocket`) or a modern browser. One
runtime dependency: `@nats-io/nats-core`, for the NATS transport.

## Installation

```bash
npm install @amniscient/detection
```

## Quick start

**Option A — the snippet.** Drop this into your app:

```ts
import { DetectionClient } from "@amniscient/detection";

const client = new DetectionClient({
  endpoint: "https://api.amniscient.com/inference",
  apiKey: "YOUR_KEY",
  organizationId: "YOUR_ORG",
});

const result = await client.detect("photo.jpg", "MODEL_ID");

console.log(result.detectionFound);                 // true / false
for (const d of result.detections) {
  console.log(d.classId, d.confidence.toFixed(2), d.bbox); // bbox: {x1,y1,x2,y2}, normalized 0-1
}
```

**Option B — a runnable script.** Save this as `detect.ts`, fill in the four
values at the top, and run it with `npx tsx detect.ts path/to/image.jpg`:

```ts
import { DetectionClient } from "@amniscient/detection";

const ENDPOINT = "https://api.amniscient.com/inference";
const API_KEY = "YOUR_KEY";
const ORG_ID = "YOUR_ORG";
const MODEL_ID = "MODEL_ID";

const imagePath = process.argv[2];
if (!imagePath) {
  console.error("Usage: npx tsx detect.ts <image_path>");
  process.exit(1);
}

const client = new DetectionClient({ endpoint: ENDPOINT, apiKey: API_KEY, organizationId: ORG_ID });

const result = await client.detect(imagePath, MODEL_ID);
console.log(`Found ${result.detections.length} objects in ${imagePath}:\n`);
for (const d of result.detections.sort((a, b) => b.confidence - a.confidence)) {
  const b = d.bbox;
  console.log(`  ${(d.confidence * 100).toFixed(0)}%  ${d.classId}`);
  console.log(`     bbox (0-1): (${b.x1.toFixed(3)}, ${b.y1.toFixed(3)}) -> (${b.x2.toFixed(3)}, ${b.y2.toFixed(3)})`);
}
```

> Top-level `await` needs `"type": "module"` in your `package.json` (or name the
> file `detect.mts`). A ready-to-run version that reads these from env vars lives
> in [`examples/basic-detection.ts`](examples/basic-detection.ts).

`detect()` accepts a **file path** (Node), raw **bytes** (`Uint8Array`/`ArrayBuffer`),
or a **`Blob`** (browser). Everything is `async` — the client is safe to construct
once and reuse.

> **Endpoint.** Use the HTTPS endpoint `https://api.amniscient.com/inference`. Your
> `apiKey` is your own application API key — validated by the API gateway on every
> request. (This SDK is HTTP-only. For NATS, use the Python SDK, `amniscient-detection` 0.5.0 or later.)

## How it works

The whole surface is `detect()`. You give it an image and a model; it returns a
`DetectionResult`. Everything in between is handled for you:

1. **Auth is validated first** — a missing API key throws `AuthenticationError`
   before any image work or network call happens. It is never skipped.
2. **The image is prepared** — loaded into bytes, optionally compressed, and
   validated.
3. **The request is sent** over HTTPS or NATS and, on a transient failure
   (network error, timeout or 5xx), retried with exponential backoff.
4. **The reply becomes a `DetectionResult`**, or a specific typed error is
   thrown (see [Error Handling](#error-handling)).

For contributors, the pieces are small and single-purpose: `client.ts` orchestrates
the flow and owns retries + error mapping; `transport.ts` moves bytes over HTTP or
NATS and nothing else; `contract.ts` is generated from `sdk/contract/contract.json`; `types.ts` holds the public shapes plus
the `filterBy*` helpers; `utils.ts` is where a reply becomes a `DetectionResult`
and where image bytes are loaded/validated; `annotate.ts` turns a payload into
draw-ready boxes; `errors.ts` holds the typed error hierarchy and the HTTP-status
mapping.

## Features

* **One call, `detect()`** — connection handling, retries, and parsing done for you
* **HTTPS or NATS** — the same calls over either; NATS keeps one WebSocket open for lower latency
* **Built on native `fetch`/`FormData`/`Blob`/`WebSocket`** — Node 22+ or a modern browser
* **Pluggable compression** — passthrough by default; supply a `compress` hook (e.g. backed by `sharp` or a `<canvas>`) to actually resize/re-encode
* **Retry logic** — exponential backoff for transient errors (network, 5xx)
* **Validated auth** — a missing/invalid API key throws `AuthenticationError`, never a silent failure
* **Fully async** — every method returns a `Promise`; fan out with `Promise.all` when you need concurrency
* **Raw mode** — `detectRaw()` returns the unparsed server payload when you need it
* **Full type hints** — `.d.ts` shipped for every export, IDE-friendly

## Usage Examples

### Different image inputs

```ts
import { readFile } from "node:fs/promises";
import { DetectionClient } from "@amniscient/detection";

const client = new DetectionClient({ endpoint: ENDPOINT, apiKey: API_KEY, organizationId: ORG_ID });

// From a file path (Node only)
const a = await client.detect("image.jpg", "MODEL_ID");

// From raw bytes (Uint8Array)
const bytes = new Uint8Array(await readFile("image.jpg"));
const b = await client.detect(bytes, "MODEL_ID");

// From an ArrayBuffer
const c = await client.detect(bytes.buffer, "MODEL_ID");

// From a Blob (browser, or Node 18+)
const blob = new Blob([bytes], { type: "image/jpeg" });
const d = await client.detect(blob, "MODEL_ID");
```

### Filtering results

```ts
import { filterByConfidence, filterByClassId, filterByTag } from "@amniscient/detection";

const result = await client.detect("image.jpg", "MODEL_ID");

// Filter by confidence threshold
const highConf = filterByConfidence(result, 0.8);

// Filter by class id
const peopleOnly = filterByClassId(result, "person", "pedestrian");

// Filter by a tag key/value
const shelfA1 = filterByTag(result, "shelf", "A1");
```

Each `filterBy*` helper returns a new `DetectionResult` — the original is left
untouched.

### Raw responses

By default `detect()` parses the server reply into a `DetectionResult`. Use
`detectRaw()` when you need the unparsed response instead — for logging/debugging,
or when downstream code already depends on the original `xywh` contract:

```ts
const payload = await client.detectRaw("image.jpg", "MODEL_ID");
// payload == { inference_id: "...", detection_found: true,
//              xywh: [{ x, y, width, height, confidence, id, tag }, ...] }
for (const det of payload.xywh as Array<Record<string, unknown>>) {
  console.log(det.id, det.confidence);
}
```

Retries, error mapping, and auth validation are unchanged — only the return
value skips parsing.

### Barcode & QR reading

Pass `detectBarcodes: true` to decode barcodes/QR codes alongside object
detection, in the same request:

```ts
const result = await client.detect("shelf.jpg", "MODEL_ID", { detectBarcodes: true });
// object detection is unchanged; barcodes ride alongside:
for (const code of result.barcodes) {
  console.log(code.type, code.value); // e.g. qr https://…  |  upc-a 0012345678905
}
```

It's off by default — pass `detectBarcodes: true` explicitly. Supported
symbologies (wire `type` values): `qr`, `upc-a`, `upc-e`, `code-39`.

Notes:

* `result.barcodes` is `[]` when `detectBarcodes` is off, no codes are found, or
  the decode step fails or times out — it never throws and never affects
  `result.detections`.
* Numeric 1D codes (UPC-A/UPC-E) decode to their expanded digit string (e.g.
  `"0012345678905"`), so don't expect an exact round-trip of a shorter printed
  value. `code-39` returns the encoded text as-is.

### Pinning a model version

Models are versioned. By default a request runs whichever version is currently
**active**, so results follow promotions automatically:

```ts
// active version — follows promotions
const result = await client.detect("shelf.jpg", "MODEL_ID");

// pinned — same weights and thresholds regardless of later promotions
const pinned = await client.detect("shelf.jpg", "MODEL_ID", { version: 3 });
```

Versions are 1-based. Pin when you need reproducible output — a benchmark, a
regression suite, or a customer held to a specific model revision. Leave it
unset to get improvements as they're promoted. The version also selects that
version's confidence/IoU thresholds and detection-item tags, not just its
weights. A version that doesn't exist throws `ModelNotFoundError`.

### Multi-tenant / per-request credentials

The client can hold one credential for its lifetime, or none at all. If you're
a **multi-tenant proxy** whose credentials arrive with each request and are
never stored server-side, build **one shared client with no stored
credentials** and pass `apiKey` / `organizationId` per `detect()` call — they
ride each request, and nothing is kept between calls:

```ts
// one shared, long-lived client — no credentials stored
const client = new DetectionClient({ endpoint: ENDPOINT });

// each request carries its own tenant's credentials
const result = await client.detect(image, "MODEL_ID", {
  apiKey: tenantKey,
  organizationId: tenantOrg,
});
```

### Over NATS (WebSocket)

For lower latency and a persistent connection, use the NATS transport. Pass
`transport: "nats"` and the NATS endpoint; the same API key works. The
organization is resolved from the key when you don't pass it, and the NATS
server confines the connection to that organization.

```ts
const client = new DetectionClient({
  endpoint: "wss://nats-server.amniscient.com",
  apiKey: "YOUR_KEY",
  transport: "nats",
});

const result = await client.detect("photo.jpg", "MODEL_ID");
const models = await client.listModels();
await client.close(); // drains the connection
```

* **One tenant per client.** NATS binds the key to the connection, so per-call
  `apiKey` / `organizationId` overrides throw a `TypeError`. Use one client per
  tenant, or HTTP for the multi-tenant proxy pattern.
* **Same errors and retries as HTTP.** A rejected key is an `AuthenticationError`;
  timeouts and 5xx replies are retried like HTTP.
* **Call `close()`** when you're done, or the open WebSocket keeps Node running.

### Validating credentials (login gate)

To accept an API key + org **before** a model is chosen (e.g. a web login
step), use `validate()` — it needs no model id and no image:

```ts
if (!(await client.validate(organizationId, apiKey))) {
  throw new Error("Invalid API key or organization");
}
```

`validate()` resolves `true`/`false` — it calls `GET /models`, so an **accepted
key gets a 200** (a real positive signal) and a **rejected key (401/403) →
`false`**. A missing route **throws** `EndpointNotAvailableError` rather than
masquerading as valid, so you never get a false "looks valid" from a route that
simply isn't there — inspect `statusCode` on the thrown `DetectionError` if you
need to tell those apart. Both `validate()` and `getDefaultModel()` are HTTP
only and authenticate through the API gateway.

### Listing models (model picker)

To let a user pick a model instead of pasting a UUID, `listModels()` returns
the org's selectable models — no model id or image needed:

```ts
const models = await client.listModels(organizationId, apiKey);
for (const m of models) {
  dropdown.add({ value: m.id, label: m.name }); // gray out non-TRAINED
  // each m: { id: "<uuid>", name: "<label>", status: "TRAINED" }
}

const dflt = await client.getDefaultModel(organizationId, apiKey); // model dict, or null
```

It returns top-level models (specialist sub-models are excluded), each with
`id`, `name`, and `status`. HTTP only, authenticated through the gateway;
`organizationId` / `apiKey` override the client's defaults.

### Putting it together — a typical flow

The methods above compose into one flow: **validate the key at login → list
models to pick one → detect on each capture.** How you hold the client depends
on your architecture.

**Single-tenant app** (one org, credentials known at startup) — one client for
its lifetime:

```ts
const client = new DetectionClient({ endpoint: ENDPOINT, apiKey, organizationId });

// 1. Login gate — reject bad credentials before showing the app
if (!(await client.validate())) {
  throw new Error("Invalid API key or organization");
}

// 2. Let the user choose a model (dropdown), instead of pasting a UUID
const models = await client.listModels();      // [{ id, name, status }, ...]
const modelId = chooseFrom(models);            // e.g. first TRAINED model

// 3. Run detection on each capture — Promise.all when you need concurrency
const results = await Promise.all(captures.map((frame) => client.detect(frame, modelId)));
results.forEach((result) => handle(result.detections));
```

**Multi-tenant proxy** (credentials arrive per request, nothing stored) —
**one shared client**, credentials passed per call:

```ts
import type { ImageInput } from "@amniscient/detection";

const client = new DetectionClient({ endpoint: ENDPOINT }); // no stored credentials

async function handleRequest(req: { org: string; key: string; image: ImageInput; modelId: string }) {
  // validate this tenant's key at their login step...
  if (!(await client.validate(req.org, req.key))) {
    return http401();
  }
  // ...then serve detections, credentials riding each call
  return client.detect(req.image, req.modelId, {
    organizationId: req.org,
    apiKey: req.key,
  });
}
```

Distinguish a real credential failure from a gateway **routing** error at the
login gate — see [Error Handling](#error-handling) for the `401` vs `403`
pattern.

### Annotating detections (draw boxes)

Turn a `/detect` response + the source image into draw-ready boxes. Pass the
**raw** payload (`detectRaw()`), unchanged:

```ts
const payload = await client.detectRaw("shelf.jpg", "MODEL_ID");
const result = await client.annotateDetections("shelf.jpg", payload); // label by name (default)

for (const b of result.boxes) {          // draw-ready specs — render however you like
  console.log(b.label, b.box);           // box = {x1,y1,x2,y2} in pixels; also id/name/confidence/color
}
```

`result.boxes` (an `AnnotationBox[]`) is the **primary product** — pixel-space,
label-resolved specs you can render in any toolkit (sharp, canvas, SVG). Options:
`{ label?: "name" | "id", color?: string, render?: boolean, names?: Record<string, string> }`
(`label` defaults to `"name"`, `color` defaults to `#00FF00`).

You can also build the same boxes directly from a payload + known image
dimensions, without a client, via the free function:

```ts
import { buildAnnotation } from "@amniscient/detection";

const boxes = buildAnnotation(payload, imgWidth, imgHeight, { label: "id" });
```

`result.toPng()` renders those boxes onto the source image and resolves with
PNG bytes — but it **needs a canvas backend** (a browser `OffscreenCanvas`/`<canvas>`,
or `createImageBitmap`). In a plain Node process it rejects with a message
telling you to render `.boxes` yourself:

```ts
// Browser (or any environment with a canvas backend):
const png = await result.toPng();   // ready-to-display PNG bytes

// Plain Node: no canvas backend — render `.boxes` yourself, e.g. with
// sharp or @napi-rs/canvas, using the pixel-space box/label/color fields.
```

**Handle the no-detections case** — an empty payload throws `NoDetectionsError`
instead of returning a blank image:

```ts
import { NoDetectionsError } from "@amniscient/detection";

try {
  const result = await client.annotateDetections(image, payload);
} catch (e) {
  if (e instanceof NoDetectionsError) {
    // nothing detected — show the original image / a "no detections" state
  } else {
    throw e;
  }
}
```

Name labels are pulled by id from the model's `detection-items` (the metadata
endpoint, a sibling of your inference endpoint), cached per model and falling
back to the id. The SDK derives that URL from an `https://…/inference` endpoint;
if yours differs, pass `metadataEndpoint` when constructing the client — or
pass a `names: { id: name }` map to `annotateDetections` to skip the network call.

## Configuration Options

`ClientOptions` passed to `new DetectionClient({ ... })`:

| Option             | Default                                        | Description                                                                                                                                  |
| ------------------ | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `endpoint`         | — (required)                                   | `https://api.amniscient.com/inference` (HTTP) or `wss://nats-server.amniscient.com` (NATS)                                                   |
| `transport`        | `"http"`                                       | `"http"` or `"nats"` (WebSocket); see [Over NATS](#over-nats-websocket)                                                                      |
| `whoamiUrl`        | known for Amniscient endpoints                 | Gateway `/whoami` used to resolve the organization over NATS when `organizationId` isn't given                                               |
| `apiKey`           | `undefined`                                    | Your application API key; can be supplied per `detect()` call instead                                                                        |
| `organizationId`   | `undefined`                                    | Organization identifier; optional (the gateway sets it from the key over HTTP, and it's resolved from the key over NATS)                     |
| `timeoutMs`        | `30000`                                        | Request timeout in milliseconds                                                                                                              |
| `maxRetries`       | `3`                                            | Retries after the first attempt for transient failures (network, timeout, 5xx); `0` = one attempt. Waits 0.25 s, 0.5 s, 1 s, … capped at 4 s |
| `detectPath`       | `/detect`                                      | Override the detect path; a `{model_id}` placeholder is substituted                                                                          |
| `compression`      | `true`                                         | Whether the `compress` hook (if any) runs before sending — a no-op passthrough when no hook is supplied                                      |
| `highCompression`  | `false`                                        | High-compression bounds (800x600) passed to the `compress` hook; overrides `jpegQuality`'s effective value                                   |
| `jpegQuality`      | `70`                                           | JPEG quality (1-100) passed to the `compress` hook                                                                                           |
| `maxDimension`     | `3840`                                         | Longest-side cap in pixels (4K UHD) passed to the `compress` hook                                                                            |
| `compress`         | `undefined` (passthrough)                      | Pluggable `ImageCompressor` — supply one (e.g. backed by `sharp` in Node or a `<canvas>` in the browser) to actually resize/re-encode        |
| `metadataEndpoint` | derived from `endpoint`'s `/inference` sibling | Base URL for `annotateDetections` object-name lookups                                                                                        |

**The compression truth:** the base SDK ships **no** image library (zero
runtime dependencies), so compression is a **passthrough by default** — bytes
are sent unchanged even with `compression: true`. This matches the Python
SDK's behavior on a base install without Pillow. Supply `compress` (an
`ImageCompressor`) to actually resize/re-encode; the `jpegQuality` /
`maxDimension` / `highCompression` knobs are then passed to it as
`CompressOptions`.

## Error Handling

Every failure surfaces as a typed subclass of `DetectionError`, each carrying
`statusCode` / `responseBody` (both `undefined` for client-side errors, i.e.
ones that never reached an HTTP response):

| Error                       | Meaning                                                                                                  |
| --------------------------- | -------------------------------------------------------------------------------------------------------- |
| `DetectionError`            | Base class for all of the below                                                                          |
| `AuthenticationError`       | Missing/invalid API key or organization ID (a genuine 401/403)                                           |
| `EndpointNotAvailableError` | The route isn't deployed on this gateway/stage (a 403 `MISSING_AUTHENTICATION_TOKEN`) — the key is valid |
| `ModelNotFoundError`        | The requested model does not exist (404)                                                                 |
| `ImageTooLargeError`        | Payload exceeds the server's size limit (413)                                                            |
| `RateLimitError`            | Too many requests (429)                                                                                  |
| `ServerError`               | Server-side (5xx) failure, after retries are exhausted (`statusCode` is set)                             |
| `ConnectionError`           | Network/timeout failure after retries are exhausted                                                      |
| `InvalidImageError`         | The image input was empty or its bytes couldn't be read                                                  |
| `NoDetectionsError`         | `annotateDetections`/`buildAnnotation` was given a payload with no detections                            |

```ts
import {
  AuthenticationError,
  ModelNotFoundError,
  ImageTooLargeError,
  RateLimitError,
  ServerError,
  ConnectionError,
} from "@amniscient/detection";

try {
  const result = await client.detect("image.jpg", "MODEL_ID");
} catch (e) {
  if (e instanceof AuthenticationError) console.error("Invalid API key or organization ID");
  else if (e instanceof ModelNotFoundError) console.error("Model not found:", e.message);
  else if (e instanceof ImageTooLargeError) console.error("Image too large:", e.message);
  else if (e instanceof RateLimitError) console.error("Rate limited — back off and retry");
  else if (e instanceof ServerError) console.error(`Server error (${e.statusCode})`);
  else if (e instanceof ConnectionError) console.error("Failed to connect to server");
  else throw e;
}
```

Transient failures (network errors, 5xx) are retried with exponential backoff
before surfacing as `ConnectionError` or `ServerError`.

**A missing route is not a rejected key.** An API Gateway returns `403
MISSING_AUTHENTICATION_TOKEN` for an **unmatched route** even when the key is
valid. The SDK recognizes that signature and throws `EndpointNotAvailableError`
(a distinct type — **not** `AuthenticationError`), so a route that simply isn't
deployed never masquerades as "bad credentials":

```ts
import { EndpointNotAvailableError, AuthenticationError } from "@amniscient/detection";

try {
  const models = await client.listModels(organizationId, apiKey); // or detect(...)
} catch (e) {
  if (e instanceof EndpointNotAvailableError) {
    // the key is fine — the route just isn't exposed on this gateway/stage
    throw new Error("Inference route not available for this environment");
  } else if (e instanceof AuthenticationError) {
    throw new Error("Invalid API key or organization"); // genuine 401/403
  } else {
    throw e;
  }
}
```

`validate()` resolves `true`/`false` for a genuine accept/reject, and **throws**
`EndpointNotAvailableError` when the route isn't deployed — it never answers a
confident `false` for a route that isn't there.

## Development

```bash
npm install
npm run build      # tsup -> dist (CJS + ESM + .d.ts)
npm test           # vitest run — 79 tests across 7 files
npm run typecheck  # tsc --noEmit
```

Run the example against a live env:

```bash
export AMNISCIENT_ENDPOINT="https://api.amniscient.com/inference"
export AMNISCIENT_API_KEY="..."; export AMNISCIENT_ORG_ID="..."; export AMNISCIENT_MODEL_ID="..."
npm run example -- path/to/image.jpg
```

**Live sandbox test** — proves a real `detect()` succeeds and a bad key is
actually rejected, end to end against the sandbox environment:

```bash
bash scripts/run-sandbox-detect.sh [image]
```

(Requires an AWS SSO refresh first; re-run it if you hit an auth error.)

More detail lives alongside this package: [`docs/USER_GUIDE.md`](docs/USER_GUIDE.md)
(task-oriented walkthrough), [`MIGRATION.md`](MIGRATION.md) (moving off direct
HTTP), [`BUILD.md`](BUILD.md) (contributors), and [`tests/README.md`](tests/README.md)
(what each test file covers).

## License

MIT