I'm trying to improve endpoint security because our support process relies heavily on administrative shares, and the current SMB access rules are too broad. I want only specific users and authorized computers to be able to access SMB on endpoint machines, while avoiding a static source-IP allowlist because users may connect from different locations.
I attempted to use Windows Connection Security Rules and a firewall rule configured to allow TCP 445 only when the connection is secure. I created matching rules for traffic in both directions, with one rule using TCP 445 as the first endpoint port and another using TCP 445 as the second endpoint port. The firewall rule on the target machine allows secure connections to local port 445.
When I try to access \Client1c$ from Client2, the connection fails. Wireshark shows ISAKMP traffic arriving from Client2, but Windows does not show an established connection under IPsec monitoring, and Event Viewer contains no useful errors despite auditing being enabled. I've tried changing endpoint order, specifying addresses, switching ports, and testing various combinations, but I still can't get an IPsec session to establish. What should I check, and is this the right approach for restricting SMB access by user and computer?
4 Answers
Before troubleshooting the IPsec rules, make sure this is actually necessary. Depending on the Windows and SMB versions involved, SMB may already provide encryption, so adding another encryption layer is not automatically a security improvement. The important control is authorization: restrict which users can authenticate to administrative shares and which computers are allowed to connect to TCP 445. A straightforward Windows Firewall rule using approved network sources may be easier to maintain if your addressing can be made predictable.
If you continue with this design, verify the complete configuration rather than just the connection security rule: IPsec policy defaults, authentication method and credentials, rule direction, profile, firewall scope, and the corresponding allow rules on both endpoints. Confirm that the firewall is actually managed by Windows and not being replaced or overridden by an antivirus or EDR product. PowerShell and the Windows IPsec monitoring tools may reveal a policy mismatch or failed authentication that the normal Event Viewer view doesn’t make obvious.
Windows IPsec negotiation can fail before SMB ever gets a chance to connect. The initiating side needs to reach the responder over IKE, commonly UDP 500, and possibly UDP 4500 when NAT traversal is involved. Check that the Windows Firewall, network ACLs, and any endpoint security product allow those negotiation packets, along with the actual protected traffic. Seeing ISAKMP packets arrive only proves that the first messages are reaching the host; it doesn’t prove that authentication, policy matching, or the security association completed.
Also watch for applications that don’t tolerate the delay while the IPsec association is being created. They may time out even when the policy eventually negotiates successfully.
Both machines are on the same subnet, and I can see the ISAKMP packets arriving. I haven’t found any EDR alert indicating that negotiation traffic is being blocked.
For a larger environment, a zero-trust access product or dedicated access gateway can isolate sensitive services from the general endpoint network and enforce identity-based access. That is often cleaner operationally, but it may be too expensive if you need a solution using only existing Windows capabilities.
The goal isn’t additional encryption. I’m using the secure-connection firewall condition primarily to require authentication, so a static IP allowlist won’t work well because the authorized users may move between computers and networks.