Transforming Your Secrets Management

Transforming Your Secrets Management

2024年 4月 9日 | Code Comments Team

Application modernization

Automation and management

Partners

Code Comments • • Transforming Your Secrets Management | Code Comments

Transforming Your Secrets Management | Code Comments

About the episode

We’ve been taught to stash our secrets. Passwords, credentials, and tokens shouldn’t be shared lest they fall into the wrong hands. When it comes to building and running large-scale systems, that thinking has to change. But maybe only a little bit.

Rosemary Wang and Steve Almy of HashiCorp share how business as usual for secrets management means putting your business at risk.

Code Comments Team Red Hat original show

Subscribe here:

Transcript

In this season of Code Comments, we're covering the well-explored paths to digital transformation. You might think there are a few secrets left to uncover. As it turns out, when your IT modernization project scales into the stratosphere, there are more secrets to know than ever. But these aren't process or development secrets to uncover and unlock the potential of your teams and infrastructure. No, these are secrets you want to keep hidden. These secrets unlock the doors to your application's proverbial kingdom. In this episode, developer advocate, Rosemary Wang and product manager Steve Almy, both at HashiCorp, explain why keeping a company's secrets safe means being a detective, a teacher and a bouncer.

There's a joke that secrets are not fun. And I think that that is very much true as a company, when you face the secrets sprawl challenge.

For the unaware, that quote Rosemary is referring to is, "Secret secrets are no fun. Secret secrets hurt someone." Managing an organization's secrets is a monumental task. While the original quote implied keeping secrets was harmful, in our case, it's sharing them that can hurt people. So first off, what is the secret? Rosemary defined them as, "Anything that gives you access to another part of the system." Think login credentials or certificates or API keys or tokens, et cetera. They're called secrets for a reason. You don't want to share them with anyone who doesn't legitimately need them. A distributed system, a cloud system, a modern system if you will, has secrets in droves. With the proliferation of components and applications needed to run a system, comes an equal proliferation of secrets to gate and grant access. And unless you've been diligent about tracking them from the start, you're going to have some sleuthing to do.

Because at this point we're now reaching a point where our systems are so complex we can't really keep track of what's going on.

There's a qualification. We can't really keep track of what's going on without automated assistance. With dozens to thousands of human users and who knows how many more interconnected application components to wrangle, secrets are all over the place and they're frequently only handled by the people who set them up. Left to their own devices, people naturally do things in their own way.

A lot of organizations have gone to cloud and experienced this organic growth of where they put their secrets. If you're using different systems, virtual machines across multiple clouds, maybe you store it in the individual Secrets Manager that is specific to that cloud provider. Other times you might choose to use Kubernetes Secrets on some other cluster somewhere. And other times you might decide to encrypt them and put them in version control and distribute them using some kind of GitOps tooling. And so the result is that you might have thousands of developers who each have a different way of doing secrets and no consolidated way to do it.

While each method may keep individual secrets well guarded, such a scattered approach can cause problems. Steve recounts a situation when a company lost access to some very important infrastructure.

I might add, even for smaller organizations, it's about de-risking the whole product space. A former employer of mine, a very small organization, had a reduction in force and reduced the person who held all of the AWS credentials. And not that they leaked, not that this individual posted them on the internet, but they didn't know how to restart systems or log into them and check their billing the next day.

In one day, they lost the ability to log into core systems. They couldn't reset them, pay for them, change configurations. What does this tell us? Proper secrets management isn't just about keeping the bad guys out. It's also about making sure the right people have access when they need it. This example with only one person having access to important infrastructure happens a lot and it happens multiple times across a single organization. Someone sets up their application components complete with their own secrets. Often they're the only ones who need access, so they don't share the secrets. But other parts of the system still depend on those components and when they go down, there's a cascading effect. If you're not on top of it, your whole system can go down and leave you scrambling trying to figure out how to get it up and running again. Let's look at another scenario. Many teams needing access to the same component, except nobody kept track of who.

And by that point we've cascaded and propagated all of these configurations and templates across tens if not twenties and thirties and hundreds of teams. And so by the time you start trying to identify who's using a secret and tracking down where it's located, you have to go, it's almost like a little hunt for where it started in the first place.

Who's using the secret? Are you using it? Did you let your friends borrow it too? How long ago was that? Rosemary has had to play detective to help customers before. She's had some unsolved mysteries land on her desk with little to go on.

Part of the challenge is that whenever I was working on a system, the first time I heard that something was wrong about rotating a secret after I did, it was a user complaint logged to the help desk. And it was fine, but I wasn't part of the help desk.

Rotating a secret is a common security practice. You update your password, you renew a certificate and so on. Standard, expected stuff. Where things get difficult is when someone needed access and automated the login, then forgot about it and the secret changed without their knowledge. Now they're getting an error message and they have no clue what's going on. That's what happened with that user complaint log. A customer service representative was handling a ticket from a customer who couldn't get into their system and only got a vague error about why.

And it took two weeks for that ticket to get circulated to my platform team. And it wasn't because it was inherently a problem, it wasn't like we didn't have systems in place to monitor it. We just didn't realize that basically three layers of backend systems had failed because one system in the very backend couldn't log in. And we had not assessed this system as being impacted by this rotation.

Without a record of who needed access to the system, some users were left out of the loop when the password was reset. And the error message wasn't clear about why they could no longer access their system. Updating secrets is a fact of digital life, but just because it's routine, it doesn't mean you should be blase about it.

Rotating a password is really helpful and you can say, "I'll rotate it and see what it impacts," and then go back and remediate. But oftentimes this will affect customers and users for weeks, if not months, and you won't find these problems until much later. It's almost something waiting to happen.

Rotating a secret, like changing a password, without telling everyone using it about the change is going to lead to some inconvenience at a minimum and major problems at worst. Relying on someone to request restored access assumes people will know what their sudden issue is. Troubleshooting could take a long time. Let's recap what we've heard.

Unmanaged secret sprawl is an issue. It creates risk because not everyone safeguards their secrets in the same way. It also creates a problem if someone is the only person with access and then is no longer part of the organization for whichever of the myriad reasons that happen every day. And not knowing who's using a secret creates problems when they're not told when that secret is rotated. These are all problems that assume secrets are being handled appropriately. As Steve and Rosemary remind us, when you don't have standard procedures to set up secrets, there are ample opportunities for them to leak.

In any number of ways and usually by accident, I have exposed it in a couple different ways before, so I'll talk about at least my ways. So for me, sometimes it's accidentally printing it out in a CI pipeline or continuous integration pipeline. The output runs and there's the password or there's the username.

Automation is a great time saver. We covered a lot of this in season two of this show. But sometimes a little bit of carelessness can lead to an automated security leak. Think forgetting to secure a secret in a pipeline or forgetting to secure it in another space.

Other times it could be that you forgot to encrypt that secret before you pushed it to version control.

So how does this kind of thing happen? Rosemary shares how working a little too quickly and without revision can lead to these unfortunate situations.

Sometimes you just copy, paste it, put in some configuration somewhere, and it's not redacted, it's not marked as sensitive. It's very easy just to put it in configuration and forget that you set it there. Part of this is that we have a tendency to distribute the secret without necessarily communicating that we distributed it.

You do the setup to finish your task and don't consider the implications of how you set it up. And sometimes they're not even secrets you set up. So if the owners of that application rotate the secret, they have no way of knowing that they need to let you know. A lot of InfoSec advice around secrets is to keep them secret, keep them safe. Don't tell anyone your password. And that's still pretty much built into the process of creating credentials.

These are things that we individually configure because secrets are so essential to how they start. And so because they're manually created in the first place, there's not much incentive to automate that kind of bootstrap.

With modern IT, cloud deployments, distributed computing, all of that changes how we need to handle secrets. It's a radical change in infrastructure and consequently needs an equivalent change in how we manage access to the system. And the new way of doing things means breaking some of the habits we've built up over the years. Some, but not all. After the break, we'll cover some of the strategies you could use to better manage your secrets.

Imagine you're trying to get past an organization's defenses. What would you call a repository of that organization's passwords, credentials and secrets? The word jackpot may come to mind. Putting all your secrets in one place seems like it might increase your risk of something terrible happening. But properly implemented, that's exactly what you'd want to do.

It seems counterintuitive to decide you're going to centralize your secrets. Because now you're saying anybody who compromises that central component that houses all of those secrets and manages them now has access to everything. But the reality is that centralizing your secrets allows you as an organization, especially as you scale across thousands of developers, to identify who's using a secret, when it's being used, you're able to audit how long it's been issued. It allows you to rotate those credentials on a life cycle.

See, if you make everyone record, store and pull their secrets from the same place, a lot of the problems we've been talking about go away. You can keep a log of who's using the secrets. Everyone gets notified when those secrets change. And eventually you can set up automations to pull those secrets from that database rather than coding them in, minimizing the chances of leaving them unencrypted in a pipeline or configuration file. While getting there isn't always easy though, it takes a lot of education and changing habits. It takes being a good teacher.

And you do have to invest in education as well because it changes the way developers are retrieving secrets or anybody in the organization is retrieving secrets. And so a lot of folks have to invest in developer enablement, but after that initial developer enablement, usually that pattern starts to get adopted and more and more people understand it.

Getting people on board and into the right habits is only part of the battle. You then have to do the work of setting up a central repository. And that can turn out to be the more difficult part.

The challenge, if you will, is once those secrets get out of control, rating them back in, pulling them back into the central place. Once you get there, much closer to utopia in terms of having control, having awareness, first of all, being able to know how to change or how to update or rotate a very specific secret to deal with a very specific unintended release of that secret.

You have to do all of that detective work we were talking about earlier. Figure out what all the secrets are, when they were created and who's using them. It's likely too much to tackle all at once. Steve advises that organizations take a staggered approach to building a centralized secrets management system.

What we found is that that really starts with having a good awareness of what secrets are being used and which services. We find customers have early successes that they can then build on over time by pulling in a couple of services at a time into a central system such as Vault. Being able to work around some of those issues with updating those secrets, proving that systems do not go down, takes a little bit of time to make sure that those rollouts happen cleanly. And then just over time, building on that success, being able to prove that some rotations are safe now to do, that services can be restarted on a very short lead time, if necessary.

You build your proof of concept, your bouncer at the speakeasy, asking for the password, teach a team how to use it and help them figure out how to get the most out of the new system. Once they've got it down and show that it works and remains secure, you can start moving on to the next team. The sprawl of secrets doesn't exactly stop. Your system keeps growing, as do the number of teams who need access to them, but now that sprawl becomes controlled growth and the benefits multiply.

There is some payback for that initial investment on the engineering development side that all of a sudden those phones don't ring, the pagers don't go off when it comes time to rotate a secret. Because it's just totally managed by the security team, the application restarts by the DevOps team, production team, and the developer can have the weekend free to enjoy dinner or whatever they're doing outside of work.

It sounds like a fairy tale ending. That's how it would be if your entire system is running on modern infrastructure. But as is often the case, especially for companies early or part way through their digital transformation, some elements are still based in traditional infrastructure. That means there's a few more considerations to keep in mind.

It's also worth noting that it's a challenge because it's a paradigm shift in the way we think about how configuration should be treated. Most applications, they don't change much configuration once it's been deployed. You could redeploy the application because you're deploying new features or you're changing the code itself, but you're not often changing a configuration. And centralized secrets management means that there's another dependency for that application to rely on, which means that application needs to be able to reconnect to a central Secrets Manager if something happens to the network.

Another shift in how you could handle your secrets. At this point, it's one of many. But it's manageable with proper planning. That's the mindset Rosemary suggests keeping in mind when tackling the secrets management side of digital transformation.

Embrace the fact that everything changes quickly. I think ephemerality, when you talk about cloud native or you talk about digital transformation, the point is that change is inevitable. Your systems must evolve with it. And you must support evolution, you're no longer supporting systems, you're supporting evolution of those systems. And change has become, I think, the primary factor in adoption of architecture. Can it change with the times?

Tech has had a lot of systems evolution over the years. Digital transformation is the latest in a long history of change. But it's different in that modern systems are now constantly changing, instead of static deployments like in the days of yore. All the support systems need to be flexible in order to keep up with the new normal and secrets management is an important part of that support system. Centralizing your secrets and implementing automations to help your team access them when needed can do a lot for your organization. Without those options, secrets can sprawl, they could be unintentionally exposed and you'll likely get a slew of help desk tickets, potentially over the course of months, whenever you need to rotate those secrets. Whatever gains you might be getting from modernizing your infrastructure could be undermined by these mounting issues. As Rosemary and Steve explained, a centralized secret system can help with those issues. It takes dedicated work to implement properly, but doing so helps you in the present and sets you up to better adapt to the unknown challenges that could come your way.

You can learn more at redhat.com/codecommentspodcast or visit redhat.com to find our guides to digital transformation. Many thanks to Rosemary Wang and Steve Almy for being our guests. Thank you for joining us. This episode was produced by Johan Philippine, Kim Huang, Caroline Creaghead and Brent Simoneaux. Our audio engineer is Mark Angley. The audio team includes Leigh Day, Stephanie Wonderlick, Mike Esser, Nick Burns, Aaron Williamson, Karen King, Jared Oates, Rachel Ertel, Carrie da Silva, Mira Cyril, Ocean Matthews, Paige Stroud, Alex Traboulsi, Boo Boo Howse and Victoria Lawton. I'm Jamie Parker and this has been Code Comments, an original podcast from Red Hat.

About the show

Code Comments

On Code Comments, we speak with experienced professionals on the challenges along the way from whiteboard to deployment.

Transforming Your Secrets Management