Import posts from another WordPress site with the REST API

When content lives on one WordPress site and needs to appear on another, copying it by hand doesn’t scale, and a one-time XML export is out of date the day after you run it. You need an import you can run again and again without creating duplicates.

That was the situation at the Louis Armstrong House Museum, which publishes its Virtual Exhibits on a separate WordPress site. I wrote the open-source Virtual Exhibit Importer to pull those posts through the WordPress REST API and create them in the museum’s main site (see the case study). Here’s how it works, and how to build a sturdier version of the same idea.

What the plugin does

The plugin adds an admin page for users with the manage_options capability, with three buttons: Start Import, Force Reimport and Delete All Exhibits. A jQuery script drives the import one step at a time through admin-ajax.php:

1. It requests /wp-json/wp/v2/posts?per_page=1 and reads the X-WP-Total header to learn how many posts exist.
2. It asks for one post per request, page by page, and advances a progress bar.
3. Each post becomes a virtual_exhibit post. The source ID is stored in an original_id meta field and the source link in original_url, which a read-only meta box shows to editors.
4. Before inserting, it looks up original_id. An existing post is skipped, or updated in Force Reimport mode, so running the import twice never creates duplicates.
5. It follows the post’s wp:featuredmedia link, downloads the image with download_url() and attaches it with media_handle_sideload().

One post per request keeps each AJAX call short and makes the progress bar accurate, which helps on hosts with strict PHP time limits. The cost is two HTTP requests per post: the post, then its media. For larger sites, or imports that run on a schedule, a WP-CLI command with bigger batches is a better fit. The rest of this article builds that version.

Fetch a page of posts

Ask for up to 100 posts per request, the API’s maximum, and add _embed=wp:featuredmedia so the image data arrives in the same response. Sorting by ID keeps pages stable if someone publishes on the source site during the import. Keep the source URL in wp-config.php, not in the plugin.

// In wp-config.php: define( 'RW_EXHIBITS_SOURCE', 'https://exhibits.example.org' );

/**
 * Fetches one page of posts from the source site.
 *
 * @return array{posts: array, total_pages: int}|WP_Error
 */
function rw_fetch_source_page( int $page, int $per_page = 50, string $modified_after = '' ): array|WP_Error {
	$args = [
		'page'     => $page,
		'per_page' => min( $per_page, 100 ), // The REST API caps per_page at 100.
		'orderby'  => 'id',
		'order'    => 'asc',
		'_embed'   => 'wp:featuredmedia',
	];
	if ( '' !== $modified_after ) {
		$args['modified_after'] = $modified_after; // ISO 8601. Needs WordPress 5.7+ on the source.
	}

	$url      = add_query_arg( array_map( 'rawurlencode', $args ), trailingslashit( RW_EXHIBITS_SOURCE ) . 'wp-json/wp/v2/posts' );
	$response = wp_safe_remote_get( $url, [ 'timeout' => 20 ] );

	if ( is_wp_error( $response ) ) {
		return $response;
	}

	$code = (int) wp_remote_retrieve_response_code( $response );
	if ( 200 !== $code ) {
		return new WP_Error( 'rw_http', sprintf( 'Source returned HTTP %d for page %d.', $code, $page ) );
	}

	$posts = json_decode( wp_remote_retrieve_body( $response ), true );
	if ( ! is_array( $posts ) ) {
		return new WP_Error( 'rw_json', sprintf( 'Invalid JSON on page %d.', $page ) );
	}

	return [
		'posts'       => $posts,
		'total_pages' => (int) wp_remote_retrieve_header( $response, 'x-wp-totalpages' ),
	];
}

Every failure path returns a WP_Error: network errors, non-200 responses and invalid JSON. wp_safe_remote_get() refuses URLs and redirects that point to local or private addresses, and the explicit timeout replaces the 5-second default. If the source posts aren’t public, send an Application Password in an Authorization header and define the credentials as constants in wp-config.php. The posts endpoint reference lists every parameter used here.

Map fields and make re-runs safe

Idempotency is the core of any importer: running it again must update what’s there, not add copies. The plugin does this with the original_id meta key. The version below does the same with one addition: it stores the source’s modified_gmt, so unchanged posts are skipped without touching the database.

function rw_find_by_source_id( int $source_id ): int {
	$ids = get_posts( [
		'post_type'      => 'virtual_exhibit',
		'post_status'    => 'any',
		'meta_key'       => '_rw_source_id',
		'meta_value'     => $source_id,
		'fields'         => 'ids',
		'posts_per_page' => 1,
		'no_found_rows'  => true,
	] );

	return $ids ? (int) $ids[0] : 0;
}

/**
 * Creates or updates the local copy of one source post.
 *
 * @return array{0: string, 1: int|WP_Error} Status ('created', 'updated', 'skipped' or 'error') and post ID or error.
 */
function rw_upsert_post( array $item, bool $force = false ): array {
	$post_id = rw_find_by_source_id( (int) $item['id'] );

	// Unchanged since the last run: nothing to write.
	if ( $post_id && ! $force && get_post_meta( $post_id, '_rw_source_modified', true ) === $item['modified_gmt'] ) {
		return [ 'skipped', $post_id ];
	}

	$date_gmt = str_replace( 'T', ' ', $item['date_gmt'] );
	$postarr  = [
		'post_type'     => 'virtual_exhibit',
		'post_status'   => 'publish',
		'post_title'    => wp_strip_all_tags( html_entity_decode( $item['title']['rendered'], ENT_QUOTES | ENT_HTML5, 'UTF-8' ) ),
		'post_name'     => sanitize_title( $item['slug'] ),
		'post_content'  => wp_kses_post( $item['content']['rendered'] ),
		'post_excerpt'  => wp_strip_all_tags( $item['excerpt']['rendered'] ),
		'post_date'     => get_date_from_gmt( $date_gmt ),
		'post_date_gmt' => $date_gmt,
		'meta_input'    => [
			'_rw_source_id'       => (int) $item['id'],
			'_rw_source_url'      => esc_url_raw( $item['link'] ),
			'_rw_source_modified' => $item['modified_gmt'],
		],
	];

	// Both functions expect slashed data. wp_update_post() keeps the fields you don't pass.
	$result = $post_id
		? wp_update_post( wp_slash( [ 'ID' => $post_id ] + $postarr ), true )
		: wp_insert_post( wp_slash( $postarr ), true );

	if ( is_wp_error( $result ) ) {
		return [ 'error', $result ];
	}

	return [ $post_id ? 'updated' : 'created', (int) $result ];
}

The underscore prefix hides the meta keys from the Custom Fields panel. If content was already imported with the plugin, keep its original_id key rather than starting a new one. A few mapping decisions deserve attention:

• Title. title.rendered contains HTML entities such as ’. Decode them so the stored title is plain text.
• Slug and date. Use the source slug and date_gmt so URLs and archive order match the original. The plugin builds the slug from the title and lets WordPress date each post at import time.
• Content. content.rendered is the HTML after blocks and shortcodes have run, not block markup. wp_kses_post() strips anything unsafe, and that includes iframe embeds. If you need the original blocks, request context=edit with an Application Password and import content.raw instead.
• Excerpt. When the author didn’t write one, excerpt.rendered is generated and may include the source theme’s “read more” text. Import it only if the source uses manual excerpts.
• Slashes. wp_insert_post() and wp_update_post() expect slashed data. Without wp_slash(), backslashes in the content disappear.

Sideload the featured image once

media_sideload_image() downloads a file, creates the attachment and, since WordPress 5.4, stores the original URL in _source_url meta. That key is exactly what you need to avoid downloading the same image again on the next run.

/**
 * Sets the featured image from the embedded media, downloading each file only once.
 *
 * @return bool|WP_Error True when a thumbnail is set, false when the source has none.
 */
function rw_sync_featured_image( int $post_id, array $item ): bool|WP_Error {
	$media = $item['_embedded']['wp:featuredmedia'][0] ?? null;
	if ( empty( $media['source_url'] ) ) {
		return false;
	}
	$source_url = esc_url_raw( $media['source_url'] );

	// media_sideload_image() stores the original URL in _source_url (WordPress 5.4+).
	$existing = get_posts( [
		'post_type'      => 'attachment',
		'post_status'    => 'inherit',
		'meta_key'       => '_source_url',
		'meta_value'     => $source_url,
		'fields'         => 'ids',
		'posts_per_page' => 1,
		'no_found_rows'  => true,
	] );

	if ( $existing ) {
		$attachment_id = (int) $existing[0];
	} else {
		require_once ABSPATH . 'wp-admin/includes/media.php';
		require_once ABSPATH . 'wp-admin/includes/file.php';
		require_once ABSPATH . 'wp-admin/includes/image.php';

		$title         = html_entity_decode( $media['title']['rendered'] ?? '', ENT_QUOTES | ENT_HTML5, 'UTF-8' );
		$attachment_id = media_sideload_image( $source_url, $post_id, sanitize_text_field( $title ), 'id' );
		if ( is_wp_error( $attachment_id ) ) {
			return $attachment_id;
		}
		// Keep the alt text written on the source site.
		update_post_meta( $attachment_id, '_wp_attachment_image_alt', sanitize_text_field( $media['alt_text'] ?? '' ) );
	}

	set_post_thumbnail( $post_id, $attachment_id );
	return true;
}

By default it accepts jpg, jpeg, jpe, png, gif and webp URLs. The image_sideload_extensions filter adds others, such as avif. The alt text comes from the source’s media object, so accessibility work done on the other site isn’t lost. Images inside the post content still point to the source site. That’s fine while the source stays online; if it will be retired, sideload those images too and rewrite their URLs in the content.

Run it with WP-CLI, in batches

An admin button is friendly, but a long import in a browser tab stops the moment someone closes it. WP-CLI runs in a terminal without the web server’s PHP time limit, and it’s easy to schedule.

if ( defined( 'WP_CLI' ) && WP_CLI ) {
	WP_CLI::add_command( 'rw exhibits import', 'rw_cli_import_exhibits', [
		'shortdesc' => 'Imports or updates posts from the source site.',
		'synopsis'  => [
			[ 'type' => 'assoc', 'name' => 'since', 'optional' => true, 'description' => 'Only posts modified after this ISO 8601 date.' ],
			[ 'type' => 'assoc', 'name' => 'per-page', 'optional' => true, 'description' => 'Posts per request, 1 to 100. Default 50.' ],
			[ 'type' => 'flag', 'name' => 'force', 'optional' => true, 'description' => 'Update posts even if unchanged.' ],
		],
	] );
}

function rw_cli_import_exhibits( array $args, array $assoc_args ): void {
	$since    = (string) ( $assoc_args['since'] ?? '' );
	$per_page = max( 1, min( 100, (int) ( $assoc_args['per-page'] ?? 50 ) ) );
	$force    = (bool) WP_CLI\Utils\get_flag_value( $assoc_args, 'force', false );
	$totals   = [ 'created' => 0, 'updated' => 0, 'skipped' => 0, 'error' => 0 ];
	$page     = 1;

	do {
		$batch = rw_fetch_source_page( $page, $per_page, $since );
		if ( is_wp_error( $batch ) ) {
			WP_CLI::error( $batch->get_error_message() ); // Stops the run. Running it again is safe.
		}

		foreach ( $batch['posts'] as $item ) {
			[ $status, $result ] = rw_upsert_post( $item, $force );
			++$totals[ $status ];

			if ( 'error' === $status ) {
				WP_CLI::warning( sprintf( 'Source post %d: %s', $item['id'], $result->get_error_message() ) );
				continue;
			}
			if ( 'skipped' !== $status ) {
				$image = rw_sync_featured_image( $result, $item );
				if ( is_wp_error( $image ) ) {
					WP_CLI::warning( sprintf( 'Image for post %d: %s', $result, $image->get_error_message() ) );
				}
			}
		}

		WP_CLI::log( sprintf( 'Page %d of %d done.', $page, max( 1, $batch['total_pages'] ) ) );
		++$page;
	} while ( $page <= $batch['total_pages'] );

	WP_CLI\Utils\format_items( 'table', [ $totals ], array_keys( $totals ) );
}

Run it as wp rw exhibits import --user=editor-name so new posts get a real author. Without the global --user flag, WP-CLI runs with no current user and new posts are saved with an author ID of 0. Because every step is idempotent, a failed run can simply be started again.

The first import runs without --since. After that, a system cron job can run the command every hour with a timestamp a little earlier than the previous run. Overlapping windows cost nothing, and modified_after needs WordPress 5.7 or later on the source site. If the host has no system cron, WP-Cron can call the same functions from an hourly event that processes one page per run. WP-Cron only fires when the site gets traffic, so treat it as “eventually”, not “on time”. The WP-CLI commands cookbook covers command registration in more depth.

What to watch out for

• Check capabilities, not only nonces. If you keep an admin-button interface like the plugin’s, call current_user_can( 'manage_options' ) in every AJAX handler. A nonce shows the request came from your page. It doesn’t show the user may run it, and a “delete all” action needs both checks.
• Deletions don’t sync. A post removed on the source stays on the destination. Compare source IDs from time to time, and move missing posts to draft rather than deleting them.
• Taxonomies need their own mapping. Match categories and tags by slug, creating terms when needed, never by ID.
• Pick a canonical URL. If both sites stay public with the same text, decide which one is canonical and declare it, or search engines will decide for you.
• Warn per item, stop per run. A bad post should log a warning and move on. A network failure should stop the run with a clear message.