I'm trying to improve endpoint security because our support process relies heavily on administrative shares, and the current SMB rules are broader than they should be. The goal is to allow SMB access only for a small set of authorized users and computers, even though the users' IP addresses may change.
I'm attempting to use Windows Connection Security Rules with the Windows Firewall's "Allow the connection if it is secure" option. On Client1, I created rules for TCP 445 in both directions:
- Endpoint 1: Any computer, TCP 445; Endpoint 2: Any computer, any TCP port
- Endpoint 1: Any computer, any TCP port; Endpoint 2: Any computer, TCP 445
I then created a firewall rule on Client1 to allow local TCP port 445 only when the connection is secure. When I try to access `\Client1c$` from Client2, the connection fails.
Wireshark shows ISAKMP traffic arriving from Client2, but Windows does not show an established connection under IPsec monitoring. There are also no obvious errors in Event Viewer, even with auditing enabled. I've tried changing endpoint addresses, reversing the ports, testing different port combinations, and configuring the rules through both the GUI and PowerShell, but I still can't get an IPsec session to establish. What should I check to determine where the negotiation is failing?
3 Answers
Before troubleshooting IPsec, verify that you actually need it for this purpose. Modern SMB can provide encryption and signing, but that doesn’t replace access-control policy. In many environments the simpler design is to restrict inbound SMB with Windows Firewall to approved management systems and enforce authorization through AD groups and administrative-share permissions. If the authorized computers are identified by changing user location or address, consider using centrally managed identity or device groups instead of trying to make IP addresses represent users.
Seeing ISAKMP packets only proves that the peers are attempting IKE negotiation; it does not mean the IPsec security association was successfully created. Check the Windows IPsec and IKEEXT operational logs on both machines, along with the main Windows Filtering Platform and firewall logs. Those usually reveal whether the failure is authentication, proposal mismatch, certificate or Kerberos validation, authorization, or a firewall drop.
Also confirm that UDP 500 and UDP 4500 are allowed between the endpoints. If NAT is involved, negotiation may move to 4500. Make sure both machines have compatible authentication methods, encryption and integrity algorithms, key lifetimes, and profile settings. A rule requiring user or computer authentication will fail if the selected credentials cannot be validated in that direction.
The clients are on the same subnet, and I can see the initial IKE traffic arriving. I’m mainly trying to figure out why there is no established session or useful error in the normal monitoring view.
Some applications also time out while the tunnel is being negotiated. Check the event channels specifically rather than relying only on the monitoring console, and test with a simple TCP connection to port 445 after the security association is established.
Be careful with the direction and meaning of the port fields. For a client connecting to `Client1:445`, the traffic initiated by the client normally has an ephemeral source port and destination port 445. The return traffic reverses those ports, so a single rule with the correct traffic direction and a matching firewall rule is usually preferable to manually building several mirrored combinations.
Confirm that the connection security rule applies to the active network profile and that no higher-priority firewall rule blocks SMB before the secure rule can match. Also check whether endpoint security software owns or filters the Windows Firewall, because it may prevent or override IPsec and port-445 rules. A useful test is to temporarily create a narrowly scoped allow rule for one known pair of computers, verify that IPsec negotiation works, and then add user or group authorization after the computer-to-computer tunnel is proven.
The intent isn’t to add another layer of encryption. I’m trying to use the secure-connection condition for authenticated authorization, since the users may move between networks and their IP addresses aren’t stable.