The Nintendo 64 (along with the GameCube and the Super Nintendo, who share a common AV connector) outputs signals in a format called "Composite Video." Composite is an old analog standard for video output. With the addition of something called a "RF Modulator" (also known as a "RF switch") these composite video signals can be received by televisions that were only able to interpret signals from RF modulated sources, like antennas or analog cable.
The composite video spec has been obsolete for many years now, and was never a particularly popular video connection for computer monitors. Also, in the intervening years, most displays switched from supporting analog signals to digital signals. In the computer world, we went from the VGA specification to the DVI specification, and then to DP and/or HDMI.
"But wait, agent," I hear you say. "I can plug a VGA->DVI adapter into my monitor, hook it up to my PC and everything works fine, and that thing costs like $5." Yes, this is because the people who designed the digital DVI specification left a bunch of pins in so that it could also carry VGA signals. The designers of HDMI did the same thing to make it backwards compatible with DVI. However, an HDMI port doesn't also output VGA signals - so you can't plug a VGA monitor into a HDMI port by chaining a HDMI->DVI and DVI->VGA adapters. It won't work.
This intelligent connector design has led to the problem you're facing. You know you can adapt certain video signals, but what you don't know is that being able to do that is very unusual. Composite video is very, very old, and there's very little interest in modern display format manufacturers to include support for it on their new, shiny HDMI/DVI/DP connectors.
Therefore, you've got to somehow convert the composite video signal to something that your modern display can handle. This is kind of an expensive process, since there's really nothing about the two signals that is compatible. You'd have to have a piece of hardware that is smart enough to understand both standards fluently. Think of it like a translator - a person who is fluent in two languages that will sit between your "composite speaking" Nintendo 64 and convert the information coming out into "HDMI speak."
These boxes can cost a pretty penny, and the better the conversion you want, the more they are likely to cost. It's hard to recommend one in particular, since what's best for you will vary, while the number of fakes and fly-by-nite sellers is extremely high. My suggestion would be to find a place with a good return policy. That way you can return it if you find it's incompatible with your hardware or produces poor image quality.
The other alternative in this is to purchase a monitor that has the electronics already integrated into it. Luckily, these are pretty easy to come by - just purchase a small television. Most televisions have integrated analog to digital converters. They tend to support RF in (so a RF modulator would be compatible) or even composite in. I have one TV that only has component (don't confuse component and composite!) in, but it's also got a mode where you can plug composite in to the same jacks as well.