Sitemap Extractor — extract URLs from sitemaps

▣ ENDPOINT 01 / 02

POST

Extract all URLs from a website's sitemaps

http://localhost:8080/apyhub/extract-sitemap-from-url-api/extract

QUICKSTARTGUIDE

Quickstart

Fetch URLs from a sitemap or website by sending the target URL in a JSON body.

curl -X POST "http://localhost:8080/apyhub/extract-sitemap-from-url-api/extract" \
  -H "apy-token: $APY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

What you'll get back

Returns a JSON object with success and data fields. On a completed extraction, data contains the extracted URLs plus optional metadata and summary information; in async mode, data contains a jobId, status, and message.

{
  "success": true,
  "data": {
    "urls": ["https://example.com/page-one"],
    "partial": false
  }
}

TRY ITLIVE · 50 ATOMS

API key

Loading your default key…

The full key is used to call the gateway and stays in this tab — never sent to orbit or saved.

About this endpoint

What it does

Extracts all URLs discovered from a website’s sitemaps. It accepts either a website URL or a direct sitemap URL, and can return the URLs immediately or start an async job depending on the selected query mode.

Query Parameter(s)

Attribute	Type	Mandatory	Description
mode	ENUM	No	Allowed value: `async`
maxUrls	Integer	No	Maximum number of URLs to return. Minimum `1`, maximum `250000`.
includeMetadata	ENUM	No	Allowed values: `true`, `1`

Request Body

Parameter	Type	Mandatory	Description
url	String	Yes	Website URL or direct sitemap URL. Must be a valid URI. If the path ends in `.xml`, `.xml.gz`, or contains `sitemap`, it is used directly; otherwise `robots.txt` is consulted and `/sitemap.xml` is tried as a fallback.

Response

Returns a JSON object with a success boolean field and a data object field. The success response is either the synchronous result payload or an async job payload, depending on the request mode.

Parameter	Type	Mandatory	Description
success	Boolean	Yes	Indicates whether the request succeeded.
data	Object	Yes	Success payload. In the synchronous shape, this object includes `meta`, `urls`, `partial`, `summary`, `warnings`, and `limitReason`. In the async shape, this object includes `jobId`, `status`, and `message`.
data.meta	Object	No	Metadata about sitemap discovery and processing.
data.meta.inputUrl	String	No	Input URL used for the extraction.
data.meta.sourceType	ENUM	No	Source type. Allowed values: `website`, `sitemap`.
data.meta.discoveryMethod	ENUM	No	Discovery method. Allowed values: `direct`, `robots_txt`, `sitemap_xml`.
data.meta.discoveredSitemaps	String Array	No	Discovered sitemap URLs.
data.meta.allProcessedSitemaps	String Array	No	All sitemap URLs processed.
data.urls	Object Array	No	Array of URL entries. Returns URL strings when `includeMetadata=false`, or objects when `includeMetadata=true`.
data.urls[].url	String	No	URL value for a sitemap entry object.
data.urls[].lastmod	String	No	Last modification date or `null`.
data.urls[].priority	Number	No	Priority value or `null`.
data.urls[].changefreq	String	No	Change frequency or `null`.
data.partial	Boolean	No	Indicates whether the URL set is partial.
data.summary	Object	No	Aggregated counts and processing metrics.
data.summary.uniqueUrls	Integer	No	Count of unique URLs.
data.summary.limitReached	Boolean	No	Indicates whether the URL limit was reached.
data.summary.totalUrlsFound	Integer	No	Total URLs found before deduplication or limiting.
data.summary.processingTimeMs	Integer	No	Processing time in milliseconds.
data.summary.duplicatesRemoved	Integer	No	Number of duplicate URLs removed.
data.summary.sitemapsProcessed	Integer	No	Number of sitemaps processed.
data.summary.invalidUrlsSkipped	Integer	No	Number of invalid URLs skipped.
data.summary.sitemapsDiscovered	Integer	No	Number of sitemaps discovered.
data.summary.urlSitemapsProcessed	Integer	No	Number of URL sitemaps processed.
data.summary.sitemapIndexesProcessed	Integer	No	Number of sitemap indexes processed.
data.warnings	String Array	No	Warning messages.
data.limitReason	ENUM	No	Limit reason. Allowed values: `MAX_URLS_REACHED`, `null`.
data.jobId	String	No	Async job identifier.
data.status	ENUM	No	Job status. Allowed values: `pending`, `running`.
data.message	String	No	Status message for the async job.

Notes

This endpoint supports an async mode when mode=async. In that case it returns immediately with data.jobId, data.status, and data.message; poll the corresponding job-check flow using the returned jobId until the status reaches a terminal state.

Query parameters

Name

Type

Description

modeOPTIONAL

string

async

—

maxUrlsOPTIONAL

integer

—

includeMetadataOPTIONAL

string

true · 1

—

Body

Name

Type

Description

bodyREQUIRED

object

—

▣ ENDPOINT 02 / 02

GET

Poll sitemap extraction job status

http://localhost:8080/apyhub/extract-sitemap-from-url-api/jobs/:jobId

QUICKSTARTGUIDE

Quickstart

Fetch the status for a sitemap extraction job by replacing jobId in the path.

curl -X GET "http://localhost:8080/apyhub/extract-sitemap-from-url-api/jobs/:jobId" \
  -H "apy-token: $APY_TOKEN"

What you'll get back

Returns a JSON object with success and data fields. success is a boolean, and data is an object whose shape depends on the job status: it may include jobId, status, and either message for pending/running, or urls and related result fields for done, or error for failed.

{
  "success": true,
  "data": {
    "jobId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "status": "pending",
    "message": "Job is still processing"
  }
}

TRY ITLIVE · 1 ATOM

API key

Loading your default key…

The full key is used to call the gateway and stays in this tab — never sent to orbit or saved.

About this endpoint

What it does

Polls the status of a sitemap extraction job using the jobId in the path. The success response is a JSON object with a success boolean and a data object whose shape depends on the job status.

Path Parameter(s)

Attribute	Type	Mandatory	Description
jobId	String	Yes	Job identifier.

Response

Returns a JSON object with a success boolean and a data object. When the job is still in progress, data includes jobId, status (pending or running), and message; when the job has failed, data includes jobId, status (failed), and an error message; when the job is done, data includes jobId, status (done) plus the extraction result fields.

Attribute	Type	Mandatory	Description
success	Boolean	Yes	Indicates whether the API call itself succeeded.
data	Object	Yes	Job status payload. The exact fields vary by status.
data.jobId	String	Yes	Job identifier.
data.status	ENUM	Yes	Job state. Allowed values: `pending`, `running`, `failed`, `done`.
data.message	String	Yes (if status is `pending` or `running`)	Status message returned while the job is still in progress.
data.error	String	Yes (if status is `failed`)	Human-readable failure reason.
data.meta	Object	No	Metadata for a completed job result. See schema for nested fields.
data.urls	String Array or Object Array	No	Extracted URLs. The schema allows an array of strings, or an array of objects with `url`, `lastmod`, `priority`, and `changefreq`.
data.partial	Boolean	No	Indicates whether the result is partial.
data.summary	Object	No	Summary of the extraction outcome. See schema for nested fields.
data.warnings	String Array	No	Warning messages returned with the result.
data.limitReason	ENUM	No	Limit reason when present. Allowed values: `MAX_URLS_REACHED`, `null`.

Path parameters

Name

Type

Description

jobIdREQUIRED

string

—

Extract Sitemap from URL API

What it does

Quickstart

What you'll get back

About this endpoint

What it does

Query Parameter(s)

Request Body

Response

Notes

Query parameters

Body

Quickstart

What you'll get back

About this endpoint

What it does

Path Parameter(s)

Response

Path parameters

Errors any endpoint can return