I've been working a lot with terraform recently, and it has been quite productive so far. Sure, there are some rough edges, but on the whole it's a really nice piece of software that lets you express all your infrastructure as code. The documentation does a fairly good job, but as with most software, there are still "gotchas" you could run into that may catch you by surprise. Here are some such things that I discovered, that I wish I knew before I started using terraform.
Use modules. Really. It doesn't matter if you think "well, I definitely wouldn't need to use this piece of code somewhere else", because you most certainly will. And in that case, it's going to be a pain to extract your configuration out into modules. To be fair, the process itself isn't that complicated, and terraform does provide you with tools to assist with the process (terraform state mv), but it's just a lot of effort that you don't have to put in if you use modules right from the beginning.
Just because terraform plan succeeded, doesn't mean terraform apply
will. I'm mostly using AWS, so this may only be specific to AWS, but there are
some operations that the AWS API doesn't allow, which plan wouldn't catch. So
it's entirely possible that if you get a green light from plan, the apply
fails somewhere in the middle because of a stupid 400 response from the AWS API.
Keep the plan and apply cycle short. Meaning, make small changes, plan
them, and apply them, to avoid having an error in the middle of a rather long
apply. But I guess this point is not specific to terraform, and applies to
software in general.
Keep terraform to strictly providing the backbone of your stack. This is a
personal opinion, but I think the work of terraform ends as soon as your
infrastructure is provisioned. Running scripts on, let's say EC2 instances
provisioned via terraform, is OK, as long as those scripts are tiny and delegate
everything else to your configuration management system. So something like
sudo apt-get install salt-minion && ./configure-salt.sh && salt-call state.apply should be OK in the user data, but more than that is just asking
for trouble.
Decide on a naming convention for your resources early. Having different styles in resource names is an eye-sore, and can be very confusing. Are you prepending your application name to each resource? Or appending? Are you using underscores? Hyphens? Spaces? Emojis? Deciding this one early enough in your project is definitely going to help.
Error messages don't always make sense. This one is not always true (hah!),
but at times I do see errors that don't completely make sense and might be
because of a stray } somewhere, or a missing parameter which may not be
obvious.
Be careful of what you're committing. Avoid committing the .tfstate and
.tfstate.backup files to git. This doesn't mean throw them away. But know
that these would contain your application secrets, so having them in your git
repository is probably not the best idea.
Lastly, and this is not really a gotcha. But terraform by itself is a very
flexible tool, so it's really up to you how you want to use it. You might find a
ton of different advice on the internet about, let's say, adding a staging
environment to your infrastructure. And they'll all probably be right. But your
setup might be different, and that's OK. As long as you know what you're doing
and have read the documentation, terraform is really a joy to work with.
NOTE: As pointed out in a comment on reddit, the .tfstate files
contain secrets only in case the provisioned resources involve secrets. So your
RDS root password would end up in the state file, but if you don't provision
any such resource, you're probably OK. The point is - be careful of what you're
committing to Git!