Where should I start building a WebSocket library in C or C++?

It sounds like a protocol implementation, but I'd check the scope with your mentor. Ask whether both client and server are required, and whether you can use existing socket or TLS libraries. That makes a big difference to the amount of work.

Start with the BSD sockets API and get a small TCP client and server exchanging bytes. Beej's Guide to Network Programming is a useful introduction. A basic HTTP client or server is a good next exercise because a WebSocket connection starts with an HTTP upgrade handshake.

In C or C++, you'll need to manage buffers and handle partial reads and writes. TCP gives you a byte stream, not neatly separated WebSocket messages, so one read might contain part of a frame or several frames.

Once TCP and the HTTP upgrade make sense, tackle RFC 6455 in pieces. Implement and test the handshake first, then frame headers, payload lengths and masking. After that, work through fragmentation, ping and pong, and the closing handshake.

Write a test for each piece as you add it. Include cases where headers and payloads arrive in chunks, as well as invalid frames. A working exchange of one short message is a good milestone, but it isn't the whole protocol.

Don't skip the masking rules. Client-to-server frames must be masked with a four-byte key, while server-to-client frames must not be masked. The payload is XORed with the repeating key, and the key travels in the frame header.

That isn't encryption or a way to hide data from the recipient. Masking helps prevent malicious browser clients from making WebSocket traffic look like chosen traffic to intermediaries. Use TLS if you need confidentiality.

If the protocol flow is still fuzzy, make a small version in Node first and use it to see when connections open, messages arrive and connections close. Then build the C or C++ version in stages. The Node API hides the byte-level work, though, so translating API calls alone won't implement the protocol.