LONDON (Reuters) - An experimental antidepressant from AstraZeneca Plc and Targacept Inc failed to meet the goal of changing a key depression rating score in the first of a series of pivotal clinical trials, in a setback for both firms.

The result, announced by the companies on Tuesday, relates to the so-called Renaissance 3 Phase III study into TC-5214, a flexible dose trial conducted in Europe.

TC-5214 -- which is viewed by analysts as a high-risk, high-reward project -- is designed to work in a novel way by modulating neuronal nicotinic receptors in the brain. Over-stimulation of these receptors is thought to be associated with depression.

AstraZeneca agreed in 2009 to pay as much as $1.24 billion for rights to the drug, including an upfront payment of $200 million, and TC-5214 has been one of the few potential bright spots in AstraZeneca's pipeline.

The setback will dent confidence in the company's line-up of new drugs, although many analysts have avoided banking on the medicine's success. AstraZeneca shares fell 1.5 percent by 0805 GMT (3:05 a.m. ET) on the news, underperforming a flat European healthcare sector.

The success of TC-5214 is more critical for Targacept, whose share price performance is linked closely to the fate of the new medicine.

The drug is the product of lengthy research at Targacept, a small U.S. drugmaker that began life as part of R.J. Reynolds Tobacco Co and has for years been using its understanding of nicotine to develop experimental treatments for depression and other psychiatric disorders.

The overall Phase III trial program consists of four studies for which all results are expected by the first half of 2012. Renaissance 3 is the first of these four studies. There is also a fifth long-term safety study that will last longer.

An AstraZeneca spokeswoman said the two partners would have to look at the results from the other three shorter trials before finalizing their intentions for TC-5214. Assuming the other data is favorable, the current plan is to file the medicine for approval in the United States in the second half of 2012, with a filing in Europe targeted for 2015.

TC-5214 is a tweaked version of mecamylamine, a drug introduced in the 1950s to treat high blood pressure. Researchers at the University of South Florida later began testing mecamylamine in the hope it would help children with Tourette's syndrome. It didn't, but research showed that it seemed to lessen depression in those children.

Targacept acquired a license to the drug and began developing it as a treatment for depression. Initial results have shown promise, but many depression drugs fail in late-stage trials, especially those that address new molecular targets.

Most new depression drugs today work by increasing the chemical serotonin, or serotonin and norepinephrine, in the brain. TC-5214 targets a different set of receptors.

(Reporting by Ben Hirschler; Editing by Hans-Juergen Peters)