Can I learn compilers deeply without a CS degree?

Yes, it is realistic, though deep knowledge will take sustained work. A degree is not a prerequisite, and a general CS degree may not go very far into compilers unless you choose a relevant course.

The tricky part of studying alone is spotting gaps. Use a university curriculum as a rough map, then check your understanding with exercises and small programs rather than just watching lectures. Programming, data structures, algorithms, discrete math and computer architecture are useful foundations. You can pick up more theory as your projects call for it.

Start small. Write a parser for arithmetic expressions, then make it evaluate those expressions. Add variables or control flow when the basic interpreter works.

After that, try translating your little language into instructions for a simple virtual machine. You will have a concrete reason to learn about syntax trees, symbol tables and code generation. A small compiler is manageable, even if production compilers are a much bigger subject.

For a bottom-up route, take a look at Nand2Tetris. It walks through building a simple computer and software for it, including an assembler and a compiler. That can make the connection between a high-level program and the machine running it much less mysterious.

Don't assume you need to finish all of mathematics before you can begin. Learn the programming basics and start experimenting, then study the formal concepts you encounter, such as grammars and parsing. Leave room to go back and fill in foundations when something isn't clicking.