Why I Switched from Recursive Copy to aws s3 sync (And What Happened When I Messed It Up on Camera)

I used to reach for a recursive copy every time I needed to pull files down from S3, even when almost nothing had changed. Switching to aws s3 sync fixed that — but not before I accidentally created a nested duplicate folder live, which turned out to be the most useful part of the whole exercise.

The problem with recursive copy

For a long time, my default move for pulling a folder's worth of objects out of an S3 bucket was a plain recursive copy. It works, but it's wasteful: a recursive copy transfers everything in scope every single time, regardless of whether the files have changed since the last run. If you're dealing with a handful of small text files, that waste doesn't matter. But once you're working with large files — image sets, video assets, build artifacts — that rarely change, re-downloading the entire folder on every run starts to cost real time and bandwidth for no reason.

That's the gap aws s3 sync is built to close. Instead of blindly copying everything, sync compares what's on the source against what's already at the destination and only transfers the files that are new or have changed. Everything that already matches gets left alone. I want to be upfront that I'm describing the general behavior here rather than the exact byte-for-byte comparison logic — AWS's own documentation is the place to go if you need the precise rules for how it decides two files are "the same," since that's the kind of detail worth confirming directly rather than taking on faith.

The promise is simple, though: same end result as a full copy, far less unnecessary transfer. I wanted to see that promise play out in practice rather than just take it on faith, so I set up a small example to test it.

Setting the stage: a folder, six files, and one deletion

To actually see sync in action, I needed a before-and-after. I started by navigating to a local folder I'd named "example," which already contained six files synced down from an S3 bucket prefix full of PNGs. Nothing fancy — just a working folder I could manipulate safely.

To simulate the exact situation sync is meant to solve — a destination that's missing something the source has — I deleted one of the six files from the local folder. That gave me a clean, controlled gap: five files present locally, six files present in the bucket, one file missing.

I'll say this plainly because it matters: I was deleting a file in a throwaway example folder I'd set up specifically for this purpose. If you're following along and trying similar commands on your own machine, be deliberate about the folder you're pointing at. Deletion commands don't ask twice, and there's no undo button waiting for you afterward. It's worth treating any delete or recursive-remove command as if it's permanent, because for practical purposes, it is.

The sync command itself

With the gap in place, I pulled up the saved command I use for this: aws s3 sync followed by a source and a destination. The source in this case is the S3 bucket prefix that holds the PNG files — essentially the bucket path acting as the folder of record. The destination is the local folder, in this case the "example" directory I'd just modified.

One detail worth calling out: the destination folder name doesn't need to match the source folder name at all. Sync doesn't care what you call your local directory — it just needs a valid path to treat as the sync target. That flexibility is convenient, but as I found out a little later in this same session, it's also exactly the kind of flexibility that can bite you if you're not paying close attention to where you're standing when you run the command.

First test: restoring exactly one missing file

This was the moment I actually wanted to see: I ran the sync command against the existing "example" folder — the one now missing a single file — and watched the output. Sync downloaded exactly one file. Not six, not zero — one. The single file that was actually missing.

That's the whole value proposition in one small, satisfying result. A recursive copy would have re-transferred all six files whether or not anything had changed, because it doesn't do this kind of comparison at all — it just copies. Sync looked at both sides, figured out the actual delta, and moved only what needed to move. For a folder of six small PNGs the difference is trivial. For a folder of large files that rarely change, this is the entire reason to prefer sync in the first place.

Second test: what happens when the destination doesn't exist yet

Restoring one missing file is the easy case. I wanted to see what sync does when there's no existing folder to compare against at all — a true first run. So I pointed the same command at a destination that didn't exist locally yet, a folder I called "folder example 2."

With nothing to compare against, sync had no partial match to work with, so it did the only sensible thing: it copied all six files. On top of that, it created the destination folder automatically — I didn't need to mkdir anything first. That's a genuinely convenient behavior worth knowing about ahead of time, because it means you can point sync at a path that doesn't exist yet and trust that it'll set things up rather than erroring out.

So the behavior scales sensibly at both ends: nothing to sync means nothing transfers, and nothing to compare against means everything transfers, folder creation included.

The live mistake: a nested folder I didn't mean to create

Here's the part of this exercise that ended up teaching me more than the planned demo did. I reran essentially the same sync command shortly after the second test — except this time, I was already sitting inside "folder example 2" when I ran it, without realizing it.

Sync did exactly what I told it to do, which was the problem. Because I was already inside the destination folder, the command created a new destination path relative to where I actually was — resulting in a folder example 2 nested inside itself. Not a bug, not a sync quirk — just me not tracking my own working directory closely enough before running the command.

I want to be honest that this wasn't planned as a teaching moment; it happened live, and I kept it in because it's a genuinely useful illustration of a very ordinary mistake. Once I noticed the nested duplicate, I identified what had happened and cleaned it up with a recursive local remove targeted at the errant subfolder. I'll flag the same caution here that I gave earlier, but it applies with extra force to recursive delete commands: know exactly what path you're pointing at before you hit enter. A recursive remove doesn't pause to ask if you're sure, and running one from the wrong working directory against the wrong target is a much easier mistake to make than it sounds like in the abstract — I'd just demonstrated that myself.

The practical lesson is simple and worth internalizing before you ever run either of these commands for real: check pwd, or otherwise confirm your current location, before running sync or any delete command — especially a recursive one — against paths you care about.

Final test: sync confirms there's nothing left to do

Once I'd cleaned up the nested folder and returned to the correct working directory, I reran the sync command one more time — this time from the right place, against the correctly matched source and destination.

Nothing transferred. Zero files. That's not a failure state, it's the payoff. It's sync confirming, correctly, that the local folder and the S3 prefix were already fully aligned, so there was nothing left to do. That's the behavior that makes sync worth reaching for on any kind of recurring basis — you can run it again and again, and it costs you almost nothing when there's no actual work to perform. A recursive copy would have re-transferred every file on that same rerun, every time, forever, regardless of whether anything had changed. Sync just checks, finds nothing new, and exits quickly. That's the entire pitch, demonstrated end to end: one missing file restored, one fresh folder populated and auto-created, one accidental nested folder diagnosed and removed, and one final run proving the two sides were truly in sync.

Conclusion

If you're still defaulting to a recursive copy out of habit, the easiest next step is to try aws s3 sync against one of your own buckets and watch what actually transfers — but before you do, take thirty seconds to confirm your current working directory and double-check the destination path you're pointing at.