I did this as an experiment when I discovered the Harman Direct Positive Paper. Fortunately, they sell 4x5" sized papers, which fit nicely into standard 4x5" film holders. As the name already implies, it's a direct positive paper, meaning that when you process it, it delivers a positive - as opposed to a negative when using standard paper. You won't need any special developer, you can use the standard paper developer.
As far as I can recall the paper's sensitivity is rated at about ISO 3. You will need rather long exposures. Plus, the paper yield a rather strong contrast, which will be difficult to handle but certainly creates an interesting effect for certain scenes.
Anyway, shooting directly on paper is not any more difficult than shooting on film. You will still need to remove the paper from the holder in the dark in order to process it, just like with film. The only difficulty with the aforementioned paper will be handling the contrast / gradiation. Other than that, very straight forward.
Wish you good luck with the experiment