The study, at the Institute of Cancer Research in London, took place on 49 men with untreatable cancer.
The drug, olaparib, had low overall success, but slowed tumour growth in 88% of patients with specific DNA mutations.
The future of cancer medicine is treating cancers by their mutated DNA rather than what part of the body they are in.
The breast cancer drug Herceptin is already used only in patients with specific mutations. Olaparib targets mutations that change the way DNA is repaired.
The trial results, published in the New England Journal of Medicine, showed the drug worked in 14 out of 16 men with such mutations.
Levels of Prostate Specific Antigen, which is produced by tumours, was more than halved and there were also significant falls in the number of prostate cancer cells detected in the blood and in the size of secondary tumours.
Patients responded to the drug for between six months and nearly a year and a half.
One of the researchers, Dr Joaquin Mateo, told the BBC News website: “It is very promising.
“Those entering the trial had an expected survival of 10 to 12 months and we have many patients on the drug for longer than a year.”
However, a larger clinical trial is needed before doctors can say if the drug extends life expectancy.
Dr Mateo added: “This is the first drug that targets specific genetically defined populations and we are going to see more and more of these coming in the next few years.”
The advantage of targeted drugs is they can be given only to those patients who will respond, which both saves money and spares patients unnecessary side effects.
Some of the patients in the study were born with mutated DNA repair genes while in others the mutation developed inside the tumour.
Professor Johann de Bono, the head of drug development at the Institute of Cancer Research said: “Our trial marks a significant step forward in the treatment of prostate cancer.
“I hope it won’t be long before we are using olaparib in the clinic to treat prostate cancer.”
However, the drugs watchdog in England – the National Institute for Health and Care Excellence – has already rejected olaparib for ovarian cancer on grounds – at £4,000 a month – of cost.
Cancer Research UK’s Dr Aine McCarthy added: “This trial is exciting because it could offer a new way to treat prostate cancer by targeting genetic mistakes in cancers that have spread.
“The hope is that this approach could help save many more lives in the future.”