Migrating a large site

A site with tens of thousands of entries, a lot of uploaded files or a strict host can outgrow a migration run from the browser. This article explains how the browser-based migration works, what its error messages mean, what to try first, and when to switch to the command line.

How a browser migration runs #

The migration screen sends the work to your server in a series of short requests, one after another, and shows the progress bar as each one returns. Each request has to finish within the time your host allows a single PHP request. GravityMigrate sizes the batches to that limit: an export writes up to 10,000 rows per step, and an import processes at most 500 entries per step. A host with a short execution time gets smaller batches. Uploaded files are fetched one per step.

A migration that runs for a long time is not stuck as long as the progress bar keeps moving. What stops it is a single request that fails.

When it stops #

“Server returned an invalid response.” means the server answered with something other than the progress data the screen expected. That is usually a PHP error or a timeout on that request, and sometimes a page served by a firewall or a security plugin. Look in the GravityKit log for the request that failed, which How to enable logging and use WP Debug in GravityKit locates. Check the server’s PHP error log at the same time, for a fatal error or a memory limit.

“The request failed. Check your connection and try again.” means the request never reached the server or the answer never came back, which is a network problem rather than a migration problem.

After a failure the screen offers Resume, which retries from the request that failed and is safe to press more than once, and Cancel, which clears the partial import from the site so you can start again. When the same step fails twice, the cause is in the logs, and fixing it comes before another attempt.

Migrations in the background #

On a site whose server can run background jobs, the wizard hands the migration to GravityKit’s job scheduler instead of driving it from the browser. The screen then reads “This is running in the background. You can leave this page and it will keep going.” The migration carries on after you close the tab, and a notice follows you across the admin screens until it finishes. Its progress also shows on the Background Jobs page under GravityKit.

Each step now runs as a scheduled job rather than as a browser request, so the per-request time limit above no longer applies. Background processing explains what a site needs for that. A site that cannot run background jobs keeps the browser-driven process described above, and the gk/gravitymigrate/background/enabled filter turns the background path off on a site that can.

What to try, in order #

1. Find the cause. A timeout on one batch is often a one-off, but a failure that repeats at the same point has a reason in the logs.
2. Check for a firewall. If the server log shows nothing, a firewall or security plugin is the usual cause. Firewall or WAF blocking GravityKit imports and exports names the request to allow for Cloudflare, ModSecurity, Wordfence and others.
3. Raise the limits, or lower the batches. Ask your host to raise PHP’s memory limit and execution time for the admin, or make each step smaller with the gk/gravitymigrate/export/batch-size and gk/gravitymigrate/import/data_chunk_sizes filters. GravityMigrate hooks has the details.
4. Move to the command line. wp gk migrate export and wp gk migrate import run the same migration without the browser and without the per-request time limit, which is the way to go for a site the browser cannot finish. Migrating Gravity Forms from the command line with WP-CLI has the commands.

Limits to know about #

• A migration file’s database dump can be up to 2 GiB. A larger one is refused with “The import bundle’s database dump file exceeded the maximum allowed size and was rejected.” The gk/gravitymigrate/import/max-dump-file-size filter raises it, and on the command line --max-dump-file-size does the same for one run.
• Each uploaded file is fetched with a limit of 100 MB and 300 seconds; the gk/gravitymigrate/import/max-download-bytes and gk/gravitymigrate/import/download-timeout filters, or the matching command line flags, change them.
• A migration file with many uploaded files takes as long as those files take to fetch from the old site, one per step. Where uploaded files go during a migration explains the fetch and its failure modes.

Related articles #

• Migrating Gravity Forms from the command line with WP-CLI
• Firewall or WAF blocking GravityKit imports and exports
• GravityMigrate hooks