▣ DATA VALIDATION · DEVELOPER TOOLS

CSV & JSON Data Cleaning and Deduplication

What it does

CSV & JSON Data Cleaning and Deduplication checks tabular data for common cleanup and deduplication issues. Send it either records as an array of objects or csv_text, then provide the columns you want treated as record keys with key_columns.

Use the built-in options to control how rows are normalised before analysis. trim_strings removes surrounding whitespace, drop_blank_rows skips empty rows, numeric_columns marks fields that should be treated as numbers, required_columns lists fields that must be present, and case_sensitive_keys controls whether key matching respects letter case.

The response schema is intentionally broad, so the service returns a structured analysis result without promising fixed field names. In practice, that makes it suitable for data validation pipelines, CSV ingestion checks, and duplicate detection before loading records into a database or analytics warehouse.

If you need to sanity-check exports from forms, spreadsheets, or ETL jobs, CSV & JSON Data Cleaning and Deduplication gives you a consistent way to inspect incoming rows and compare them against the key columns you care about.

▣ ENDPOINT 01 / 02

POST

Analyze CSV/JSON

http://localhost:8080/giacomo-petrioli/analyze/analyze

QUICKSTARTGUIDE

Quickstart

Send a small set of records to analyze for duplicates, using the required JSON body.

curl -X POST "http://localhost:8080/giacomo-petrioli/analyze/analyze" \
  -H "apy-token: $APY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "records": [
      {
        "id": "rec-001",
        "name": "Ava Patel",
        "email": "[email protected]"
      },
      {
        "id": "rec-002",
        "name": "Ava Patel",
        "email": "[email protected]"
      }
    ],
    "key_columns": ["email"]
  }'

What you'll get back

Returns a JSON object. The schema does not declare any top-level response fields, so the exact shape is not specified here.

{
  "summary": {
    "rows_input": 2,
    "rows_output": 1,
    "columns": [
      "email",
      "id",
      "name"
    ],
    "duplicates_removed": 1,
    "blank_rows_removed": 0,
    "quality_issues": 1
  },
  "duplicate_rows": [
    {
      "row_index": 1,
      "duplicate_of": 0,
      "key": {
        "email": "[email protected]"
      }
    }
  ],
  "missing_required": {},
  "numeric_errors": {},
  "null_counts": {
    "email": 0,
    "id": 0,
    "name": 0
  },
  "type_profiles": {
    "email": {
      "string": 1
    },
    "id": {
      "string": 1
    },
    "name": {
      "string": 1
    }
  },
  "cleaned_records": [
    {
      "email": "[email protected]",
      "id": "rec-001",
      "name": "Ava Patel"
    }
  ],
  "content_sha256": "d64b9099ea25293222205400a7e797d03852cbee43aef24831d1d3b7c77e6107"
}

TRY ITLIVE · 10 ATOMS

API key

Loading your default key…

The full key is used to call the gateway and stays in this tab — never sent to orbit or saved.

About this endpoint

What it does

Analyzes the submitted dataset from either structured records or raw csv_text input. The request can also include column lists and normalization flags that influence how the analysis is performed.

Request Body

Parameter	Type	Mandatory	Description
records	Object Array	No	Array of record objects to analyze. Can be `null`.
csv_text	String	No	Raw CSV input text to analyze. Can be `null`.
key_columns	String Array	No	Column names to use as key columns.
trim_strings	Boolean	No	Trims string values before processing. Default: `true`.
drop_blank_rows	Boolean	No	Drops blank rows before processing. Default: `true`.
numeric_columns	String Array	No	Column names to treat as numeric columns.
required_columns	String Array	No	Column names that are required.
case_sensitive_keys	Boolean	No	Treats key column matching as case-sensitive. Default: `false`.

Response

Returns a JSON object with an additional-properties response shape; the schema does not define any fixed top-level fields. The concrete success status code is not declared in the schema.

Body

Name

Type

Description

bodyREQUIRED

object

—

▣ ENDPOINT 02 / 02

GET

Capabilities

http://localhost:8080/giacomo-petrioli/analyze/capabilities

QUICKSTARTGUIDE

Quickstart

Fetch the service capabilities with a simple GET request.

curl -X GET "http://localhost:8080/giacomo-petrioli/analyze/capabilities" \
  -H "apy-token: $APY_TOKEN"

What you'll get back

Returns a JSON object with additional top-level properties. The schema does not define fixed fields, so the response shape may vary.

{
  "service": "Data Quality & Deduplication API",
  "version": "1.0.1",
  "inputs": [
    "JSON records",
    "CSV text"
  ],
  "features": [
    "configurable key-based deduplication",
    "whitespace normalization",
    "blank-row removal",
    "required-field checks",
    "numeric-field validation",
    "null and type profiles",
    "SHA-256 cleaned-content receipt"
  ],
  "limits": {
    "records": 10000,
    "columns": 200,
    "body_bytes": 5242880
  },
  "data_handling": {
    "application_storage": "none",
    "executes_input": false,
    "follows_input_urls": false,
    "note": "The hosting provider and marketplace proxy may maintain their own metadata logs."
  }
}

TRY ITLIVE · 1 ATOM

API key

Loading your default key…

The full key is used to call the gateway and stays in this tab — never sent to orbit or saved.

About this endpoint

What it does

Retrieves the capabilities response for this endpoint. The output is a JSON object, and the schema allows additional properties, so the exact response fields are not fixed by the schema.

Response

Returns a JSON object. The response schema does not define any fixed top-level fields; it only specifies that the payload is an object with additionalProperties: true.

Parameters

No parameters.

▣ COMMON ERRORS

Errors any endpoint can return

400bad_request

Required parameter missing or malformed body.

401unauthorized

API key missing, revoked, or not authorized for this service.

429rate_limited

Your plan's per-second rate exceeded. Retry with exponential backoff.

503upstream_busy

Backend temporarily unavailable. Try again in a few seconds.