How can I determine if the lens I have is compatible with the camera I have?

I just bought a used nikon F801 with a 50mm Nikkor f1.4 lens and SB-28 flash. However I've determined that there's a problem with it not working in program mode. It seems like it always takes the pictures at the smallest aperture even when the camera says something else, or in other words, the camera isn't controlling the aperture. It still works in aperture priority, and it also still works with TTL flash, because the flash just keeps flashing until proper exposure even though the lens is stopped all the way down. However most photos taken in program mode are way underexposed.

So since the aperture isn't working in program mode, either the lens is broken, the camera is broken, or the two are incompatible with each other.

If you flick the aperture return lever on the back, does the aperture open and close swiftly or is it sluggish?

It works fine, very smooth. I don't think there is anything wrong with the lens; it must be the camera body. We are going to try a different nikon AF lens tomorrow. I'm pretty sure it's not going to work either. But could it be a compatibility thing? I've heard not all nikon lenses fit all nikon cameras (I'm used to pentax).

Here's a link to a lens compatibility chart for the N8008s (same camera as yours, except for spot metering, I believe): http://www.mir.com.my/rb/photography/companies/nikon/htmls/models/htmls/Nikonf801sN8008s.htm

I have one of these cameras- while almost all Nikon lenses will mount, not all AF lenses will give every feature you may expect. Here's as good a guide as any-
http://www.kenrockwell.com/nikon/compatibility-lens.htm

You also need to unlock the aperture ring to use the AF lens in aperture-priority mode or lock it for full program.

Yes I had it locked and in program or program_H when it was malfunctioning. The only time it worked was when I was using TTL flash, because the lens stays fully stopped down, regardless of what the camera program mode indicates.

edit: I checked the chart and it says that the lens is compatible (it has five contacts and a screw-slotted thing). So apparently it's that the camera is broken.

How does it feel when you press the DOF preview button. do you feel the resistance from the spring, or is it "sloppy" and does it sound right?
Kind regards

The DOF preview button seems to operate fine. Nothing strange.