Terraform AWS provider · troubleshooting

bufio.Scanner: token too long

Terraform AWS provider 6.69.0 fails any API call with a line over 64 KB: large IAM managed policies, Lambda layer uploads, API Gateway templates. Upgrade to 6.69.1 or later with terraform init -upgrade. In your own Go code the same message means a bufio.Scanner line exceeded its 64 KB default.

First published 21 Oct 2026

New to this? Read what is actually happening first, then come back to the steps.

On this page · 10 sections

Before you start

The failing plan or apply output, which names the AWS operation.
The .terraform.lock.hcl file and the required_providers block, to see which provider version is selected.
Network access to the Terraform registry to download the fixed provider.

Run this first

step 1 of 2

terraform providers
grep -A2 'hashicorp/aws' .terraform.lock.hcl

Look for 6.69.0 in the lock file

Shows the constraint each module sets and the exact version the lock file selected.

Work out which cause you have

A few questions to narrow the list. Every answer ends in a command that confirms or rules the cause out — this cannot see your cluster, so nothing here is a certainty until you have checked.

Narrow it down

0 answered · nothing is sent anywhere

Where does the error come from?

Cause space

2 of 2 still possible

The AWS provider is version 6.69.0Common
Your own Go program reads a line longer than 64 KBOccasional

Nothing ruled out yet. Answer the question above and the branches your answer eliminates will strike through here.

Check it with a tool

Amazon IAM explained

Or diagnose it manually

In this order. The first command usually contains the whole answer.

Step 1
```
terraform providers
grep -A2 'hashicorp/aws' .terraform.lock.hcl
```
Shows the constraint each module sets and the exact version the lock file selected.
Look for 6.69.0 in the lock file
Step 2
```
terraform init -upgrade && terraform plan
```
Selects the newest allowed version and repeats the failing operation.
Look for The plan completes; init reports hashicorp/aws v6.69.1 or later

Every cause, and how to fix it

Ordered by how often each one turns out to be the answer.

The AWS provider is version 6.69.0

Common

A run of terraform init -upgrade, or a version constraint such as ~> 6.0 without a lock file, selected 6.69.0, the only release with the regression.

Confirm

terraform version
grep -A2 'registry.terraform.io/hashicorp/aws' .terraform.lock.hcl

hashicorp/aws v6.69.0, or version = "6.69.0" in the lock file

Fix

Upgrade the provider: run terraform init -upgrade, which selects the newest version your constraints allow (6.69.1 or later), and commit the updated .terraform.lock.hcl.
If your constraint pins exactly 6.69.0, change it, for example to version = ">= 6.44.0, != 6.69.0", then run terraform init -upgrade.
In CI, make sure the job uses the committed lock file, so a cached 6.69.0 cannot come back.

terraform init -upgrade

Then commit .terraform.lock.hcl. OpenTofu: tofu init -upgrade.

Your own Go program reads a line longer than 64 KB

Occasional

The message is Go's bufio.ErrTooLong. Any program that reads input with bufio.Scanner and its default buffer stops at the first line over 64 KB: minified JSON, a long log line, a base64 blob.

Confirm

awk '{ if (length($0) > 65536) print FILENAME": line "NR" is "length($0)" characters" }' INPUT_FILE

A line longer than 65536 characters in the input

Fix

Give the scanner a larger limit before the first Scan: sc.Buffer(make([]byte, 0, 64*1024), 10*1024*1024) allows lines up to 10 MB.
For input with no natural line limit, read it with bufio.Reader.ReadString or json.Decoder instead of a line scanner.

Understanding it properly

Skip this if you are mid-incident — the working part of the page is above. Worth reading afterwards, because understanding the mechanism is what stops the next one.

In plain terms

Version 6.69.0 of the AWS provider changed how it logs what it sends to and receives from AWS. The new code read each message line by line with a Go tool that refuses any line over 64 KB. Some AWS answers are one very long line, such as the ReadOnlyAccess policy document, so those calls failed with this message. The next day's release, 6.69.1, fixed it. Nothing is wrong with your configuration.

Words used on this page

bufio.Scanner: Go's standard line reader. By default it refuses any single line longer than 64 KB and returns this exact error.
Dependency lock file: .terraform.lock.hcl records the exact provider version; init keeps using it until you run terraform init -upgrade.
decomposing response / request: The AWS SDK step that processes a message; the error arrived there, after AWS had already answered with status 200.

What is actually happeningadvanced

6.69.0 (14 Oct 2026) picked up a change in aws-sdk-go-base that logs request and response bodies by reading them line by line with Go's bufio.Scanner. Its default limit is 64 KB per line (bufio.MaxScanTokenSize), and a longer line returns bufio.Scanner: token too long, which failed the whole API call.

Reports came from reading large AWS-managed IAM policies (arn:aws:iam::aws:policy/ReadOnlyAccess through GetPolicyVersion), publishing Lambda layer versions from zip files, and API Gateway methods and integrations with request templates over 64 KB. Any operation with a long enough line could fail the same way.

6.69.1 (15 Oct 2026) updated aws-sdk-go-base and fixed it for all of them (#50501). 6.68.0 and earlier never had it.

The status code in the message is 200: AWS answered normally, and the failure is entirely inside the provider.

How to tell this is your problemsignals

Where	What you see
terraform plan or apply	Error: reading IAM Policy (arn:aws:iam::aws:policy/ReadOnlyAccess) default version: operation error IAM: GetPolicyVersion, https response error StatusCode: 200, … decomposing response: bufio.Scanner: token too long
terraform apply of aws_lambda_layer_version	The PublishLayerVersion call fails with decomposing request: bufio.Scanner: token too long
terraform init output	Installed hashicorp/aws v6.69.0

How to know it is actually fixed

terraform version lists hashicorp/aws v6.69.1 or later.
The plan that failed now completes, with no changes if nothing else changed.

Stopping it happening again

Commit .terraform.lock.hcl and upgrade providers on purpose, in a pull request whose plan you read, rather than with -upgrade in every CI run.
Upgrade a day or two after a provider release on production stacks; regressions like this one are usually fixed within days.

Security considerations

Do not work around this by replacing an AWS-managed policy with a hand-copied, trimmed version: it silently stops tracking AWS's updates to that policy.
Review the provider upgrade like any other: commit the new .terraform.lock.hcl hashes rather than editing the lock file by hand.

Sources

Behaviour described here is drawn from official documentation. Where a figure could not be confirmed on an official page it is attributed in the text rather than stated as canonical.

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it