Radmin doesn't work with a modem directly. To use a modem-to-modem connection you need to configure Dial-Up networking on both the server and client computers.
Radmin uses a TCP/IP connection between computers. So, you need to set up this service on both computers.