Introducing simple and secure egress policies by hostname in Cloudflare’s SASE platform

Cloudflare’s SASE platform is on a mission to strengthen our platform-wide support for hostname- and domain-based policies. This mission is being driven by enthusiastic demands from our customers, and boosted along the way by several interesting engineering challenges. Today, we’re taking a deep dive into the first milestone of this mission, which we recently released in open beta: egress policies by hostname, domain, content category, and application. Let’s dive right in! 

Customers use our egress policies to control how their organization’s Internet traffic connects to external services. An egress policy allows a customer to control the source IP address their traffic uses, as well as the geographic location that their traffic uses to egress onto the public Internet. Control of the source IP address is especially useful when accessing external services that apply policies to traffic based on source IPs, using IP Access Control Lists (ACLs). Some services use IP ACLs because they improve security, while others use them because they are explicitly required by regulation or compliance frameworks. 

(That said, it’s important to clarify that we do not recommend relying on IP ACLs as the only security mechanism used to gate access to a resource. Instead, IP ACLs should be used in combination with strong authentication like Single Sign On (SSO), Multi Factor Authentication (MFA), passkeys, etc.).

Let’s make the use case for egress policies more concrete with an example. 

Imagine that Acme Co is a company that has purchased its own dedicated egress IP address 203.0.113.9 from Cloudflare. Meanwhile, imagine a regulated banking application (https://bank.example.com) that only grants access to the corporate account for Acme Co when traffic originates from source IP address 203.0.113.9. Any traffic with a different source IP will be prevented from accessing Acme Co’s corporate account. In this way, the banking application uses IP ACLs to ensure that only employees from Acme Co can access Acme Co’s corporate account. 

Continuing our example, suppose that Acme Co wants to ensure that the banking application is off limits to all of its employees except those on its finance team. To accomplish this, Acme Co wants to write an egress policy that allows members of the finance team to egress from 203.0.113.9 when accessing https://bank.example.com, but employees outside of finance will not egress from 203.0.113.9 when attempting to access https://bank.example.com.  

As shown in the figure above, the combination of the banking application’s IP ACLs and Acme Co’s egress policies ensures that https://bank.example.com is only accessible to its finance employees at Acme Co. 

While this all sounds great, until now, this scenario was fairly difficult to achieve on Cloudflare’s SASE platform. While we have long supported egress policies by user groups and other user attributes, we did not support writing egress policies by hostname. Instead, customers had to resort to writing egress policies by destination IP addresses.

To understand why customers have been clamoring for egress policies by hostname, let’s return to our example: 

In our example, Acme Co wants to write a policy that allows only the finance team to access https://bank.example.com. In the past, in the absence of egress policies by hostname, Acme Co would need to write its egress policy in terms of the destination IP address of the banking application. 

But how does Acme Co know the destination IP address of this banking application? The first problem is that the destination IP address belongs to an external service that is not controlled by Acme Co, and the second problem is that this IP address could change frequently, especially if the banking application uses ephemeral infrastructure or sits behind a reverse proxy or CDN. Keeping up with changes to the destination IP address of an external service led some of our customers to write their own homegrown scripts that continuously update destination IP Lists which are then fed to our egress policies using Cloudflare’s API.

With this new feature, we do away with all these complications and simply allow our customers to write egress policies by hostname. 

Before we continue, we should note that this new feature also supports writing egress policies by:

• Domain: For example, we can now write an egress policy for *.bank.example.com, rather than an individual policy for each hostname (bank.example.com, app.acmeco.bank.example.com, auth.bank.example.com, etc.)

• Category: For example, we can now write a single egress policy to control the egress IP address that employees use when accessing a site in the Cryptocurrency content category, rather than an individual policy for every Cryptocurrency website.

• Application: For example, we can write a single egress policy for Slack, without needing to know all the different host and domain names (e.g. app.slack.com, slack.com, acmeco.slack.com, slack-edge.com) that Slack uses to serve its application.

Here’s an example of writing an egress policy by application:

^{A view of the Cloudflare dashboard showing how to write an egress policy for a set of users and Applications. The policy is applied to users in the “finance” user group when accessing Applications including Microsoft Team, Slack, BambooHR and Progressive, and it determines the source IP that traffic uses when it egresses to the public Internet.}

Now let’s get into the engineering challenges behind this feature.

Egress polices are part of Cloudflare Gateway. Cloudflare Gateway is a Secure Web Gateway (SWG) that operates as both a layer 4 (L4) and layer 7 (L7) proxy. In other words, Cloudflare Gateway intercepts traffic by inspecting it at the transport layer (layer 4, including TCP and UDP), as well as at the application layer (layer 7, including HTTP).

The problem is that egress policies must necessarily be evaluated at layer 4, rather than at layer 7. Why? Because egress policies are used to select the source IP address for network traffic, and Cloudflare Gateway must select the source IP address for traffic before it creates the connection to the external service bank.example.com. If Gateway changes the source IP address in the middle of the connection, the connection will be broken. Therefore, Gateway must apply egress policies before it sends the very first packet in the connection. For instance, Gateway must apply egress policies before it sends the TCP SYN, which of course happens well before it sends any layer 7 information (e.g. HTTP). (See here for more information on Gateway’s order of enforcement for its policies.)

The bottom line is that Gateway has no other information to use when applying the egress policy, other than the information in the IP header and the L4 (e.g. TCP) header of an IP packet. As you can see for the TCP/IPv4 packet below, a destination hostname is not part of the IP or TCP header in a packet. That’s why we previously were not able to support egress policies by hostname, and instead required administrators to write egress policies by destination IP address.

We took advantage of the fact that Cloudflare Gateway also operates its own DNS resolver. Every time an end user wants to access a destination hostname, the Gateway resolver first receives a DNS query for that hostname before sending its network traffic to the destination hostname. 

To support egress policies by hostname, Gateway associates the DNS query for the hostname with the IP address  and TCP/UDP information in the network connection to the hostname. Specifically, Gateway will map the destination IP address in the end-user’s network connection to the hostname in the DNS query using a “synthetic IP” mechanism that is best explained using a diagram:

Let’s walk through the flow:

1. When the end user makes a DNS query for bank.example.com, that DNS query is sent to the Gateway resolver.

2. The Gateway resolver does a public DNS lookup to associate bank.example.com to its destination IP address, which is 96.7.128.198.

3. However, the Gateway resolver will not respond to the DNS query using the real destination IP 96.7.128.198. Instead, it responds with an initial resolved IP address of 100.80.10.10. This is not the real IP address for bank.example.com; instead, it acts as a tag that allows Gateway to identify network traffic destined to bank.example.com.  The initial resolved IP is randomly selected and temporarily assigned from one of the two IP address ranges below, which correspond to the Carrier Grade Network Address Translation (CGNAT) IP address spaces as defined in RFC 6598 and RFC 6264, respectively.

IPv4: 100.80.0.0/16

IPv6: 2606:4700:0cf1:4000::/64 

4. Gateway has now associated the initial resolved IP address 100.80.10.10, with the hostname bank.example.com. Thus, when Gateway now sees network traffic to destination IP address 100.80.10.10, Gateway recognizes it and applies the egress policy for bank.example.com. 

5. After applying the egress policy, Gateway will rewrite the initially resolved address IP 100.80.10.10, on the network traffic with the actual IP address 96.7.128.198 for bank.example.com, and send it out the egress IP address so that it can reach its destination.

The network traffic now has the correct destination IP address, and egresses according to the policy for bank.example.com, and all is well! 

So far, we’ve seen how the mechanism works with individual hostnames (i.e. Fully Qualified Domain Names (FQDNs) like bank.example.com). What about egress policies for domains and subdomains like *.bank.example.com? What about content categories and applications, which are essentially sets of hostnames grouped together?

We are able to support these use cases because (returning to our example above) Gateway temporarily assigns the initial resolved IP address 100.80.10.10 to the hostname bank.example.com for a short period of time. After this short time period, the initial resolved IP address is released and returned into the pool of available addresses (in 100.80.0.0/16), where it can be assigned to another hostname in the future.

In other words, we use a random dynamic assignment of initial resolved IP addresses, rather than statically associating a single initial resolved IP address with a single hostname. The result is that we can apply IPv4 egress policies to a very large number of hostnames, rather than being limited by the 65,536 IP addresses available in the 100.80.0.0/16 IPv4 address block.

Randomly assigning the initial resolved IP address also means that we can apply a single egress policy for a wildcard like *.bank.example.com to any traffic we happen to come across, such as traffic for acmeco.bank.example.com or auth.bank.example.com. A static mapping would require the customer to write a different policy for each individual hostname, which is clunkier and more difficult to manage.

Thus, by using dynamic assignments of initial resolved IP addresses, we simplify our customers’ egress policies and all is well!

Actually, not quite. There’s one other problem we need to solve.

Cloudflare has an extensive global network, with servers running our software stack in over 330 cities in 125 countries. Our architecture is such that sharing strongly-consistent storage across those servers (even within a single data center) comes with some performance and reliability costs. For this reason, we decided to build this feature under the assumption that state could not be shared between any Cloudflare servers, even servers in the same data center.

This assumption created an interesting challenge for us. Let’s see why.

Returning to our running example, suppose that the end user’s DNS traffic lands on one Cloudflare server while the end user’s network traffic lands on a different Cloudflare server. Those servers do not share state.  Thus, it’s not possible to associate the mapping from hostname to its actual destination IP address (bank.example.com = 96.7.128.198) which was obtained from the DNS traffic, with the initial resolved IP that is used by the network traffic (i.e. 100.80.10.10). Our mechanism would break down and traffic would be dropped, as shown in the figure below.

We solve this problem by ensuring that DNS traffic and network traffic land on the same Cloudflare server. In particular, we require DNS traffic to go into the same tunnel as network traffic so that both traffic flows land on the same Cloudflare server. For this reason, egress policies by hostname are only supported when end users connect to the Cloudflare network using one of the following on-ramps:

We are actively working to expand support of this feature to more onramps. 

There’s a lot more coming. Besides expanding support for more onramps, we also plan to extend this support to hostname-based rulesets in more parts of Cloudflare’s SASE. Stand by for more updates from us on this topic. All of these new features will rely on the “initial resolved IP” mechanism that we described above, empowering our customers to simplify their rulesets and enforce tighter security policies in their Cloudflare SASE deployments.

Don’t wait to gain granular control over your network traffic: log in to your Cloudflare dashboard to explore the beta release of egress policies by hostname / domain / category / application and bolster your security strategy with Cloudflare SASE.