Fix: RequestTimeTooSkewed (AWS)

If an AWS SDK or CLI call fails with RequestTimeTooSkewed, the problem is not your credentials or permissions; it is your clock. AWS signs and validates every request with a timestamp, and when the client’s clock differs from AWS time by more than about 15 minutes, the signature is rejected before the request is even authorized. The calls start working again the moment the host clock is back in sync, so the real fix is reliable time synchronization on the machine making the request.

An error occurred (RequestTimeTooSkewed) when calling the PutObject operation: The difference between the request time and the current time is too large.

What this error means

AWS Signature Version 4 embeds the current time in each request and signs it. On the server side AWS recomputes the signature and checks the embedded timestamp against its own clock. If the two clocks are too far apart, more than roughly 15 minutes, AWS assumes the request could be a replay and refuses it with RequestTimeTooSkewed. Because the timestamp is baked into the signature, a skewed clock does not just look suspicious; it produces a signature that cannot match, so the request fails deterministically.

This is a host clock problem wearing an AWS error message. Fix the time and the API calls succeed with no other change.

Common causes

• A VM or laptop resumed from sleep, hibernation, or a snapshot and its clock fell behind real time.
• A container is reading a host or node clock that is itself out of sync.
• No time synchronization daemon is running, so drift accumulates until it crosses the 15 minute threshold.
• An IoT or edge device with no reliable network time source and a drifting hardware clock.
• The time zone was changed instead of the actual clock, or the hardware clock is set to local time while software expects UTC.
• A firewall blocks outbound NTP (UDP 123), so the sync daemon runs but never corrects the clock.

How to fix it

STABILIZE first. If a deploy or job is blocked right now, correct the clock immediately so the calls succeed, then set up ongoing sync so it stays correct. On a host with systemd-timesyncd:

sudo timedatectl set-ntp true
sudo systemctl restart systemd-timesyncd

DIAGNOSE the skew so you know how far off you are and whether sync is even running. First compare the host’s UTC time to reality:

date -u

Then check whether the clock is considered synchronized and which service manages it:

timedatectl status

Look at the System clock synchronized and NTP service lines. If synchronization is inactive, or the offset is large, that is the cause. If you run chrony, confirm it is actually tracking a source and see the current offset:

chronyc tracking

For a container showing wrong time, do not debug inside it; check the host or node clock instead, because the container inherits it:

docker run --rm alpine date -u

If that time is wrong, fix the host, not the image.

FIX by establishing reliable synchronization on the host.

1. On EC2, use the Amazon Time Sync Service, which is available locally with no internet route. Point chrony at the link-local address:

# /etc/chrony/chrony.conf
server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4

Then restart and force an immediate correction:

sudo systemctl restart chrony
sudo chronyc makestep

2. On a general Linux host without chrony, enable systemd-timesyncd and confirm it steps the clock:

sudo timedatectl set-ntp true
timedatectl show-timesync --property=NTPMessage

3. If the hardware clock is set to local time rather than UTC, which causes a fixed offset, set it to UTC:

sudo timedatectl set-local-rtc 0

4. If a firewall is blocking NTP, allow outbound UDP 123 (or reach the EC2 time service, which does not need internet egress). Verify the daemon can actually reach a source with chronyc sources before assuming it is fixed.

After correcting the clock, retry the AWS call; no credential or SDK change is required.

How to prevent it

• Run a time sync daemon (chrony or systemd-timesyncd) on every host and node, and monitor that it is actually synchronized, not just installed.
• On EC2, prefer the Amazon Time Sync Service at 169.254.169.123 so correction is fast and does not depend on internet egress.
• Configure the sync daemon to step the clock on resume so VMs that sleep or restore from snapshot self-correct immediately.
• Keep hardware clocks on UTC, not local time, to avoid a fixed offset.
• Ensure firewalls permit the NTP path the daemon uses, and alert if the clock offset exceeds a few seconds.
• For edge and IoT fleets, provide a reachable time source and monitor per-device offset centrally.

When to call a senior engineer

Call for help when the clock keeps drifting despite a running sync daemon, when skew appears across a fleet and you cannot pin the source, or when containers show correct host time yet still hit RequestTimeTooSkewed, which points at a namespaced or virtualized clock issue. Erzon engineers can trace the time path from hardware clock through the sync daemon to the running process, close the network gaps that starve NTP, and leave your fleet with clocks that stay inside AWS’s tolerance so signing never fails on time again.