(Reuters) - The beginning of the end of the Federal Reserve's massive bond-buying program might come sooner than many investors think if recent gains in the U.S. labor market do not prove fleeting.

Much will depend on how economic data, which has given mixed signals for growth prospects, develops over the next few months. Reports on job growth in particular will go a long way in helping Fed officials determine whether the time is right to trim the pace of their $85 billion in monthly purchases.

The marked improvement in the labor market since the U.S. central bank began its third round of quantitative easing, or QE3, has added an edge to calls by some policy hawks to dial down the stimulus. The roughly 50 percent jump in monthly job creation since the program began has even won renewed support from centrists, raising at least some chance the Fed could ratchet back its buying as early as next month.

"We could reduce somewhat the pace of our securities purchases, perhaps as early as this summer," San Francisco Federal Reserve Bank President John Williams said on Thursday, adding that his view is that summer begins in mid-June.

The central bank next meets to debate policy on June 18-19.

The Fed's balance sheet has swelled to some $3.3 trillion and officials have been debating whether this risks igniting future inflation or blowing up asset bubbles, even as they seek to help a tepid economic recovery.

Chairman Ben Bernanke and other top Fed officials have increasingly stressed that any change to the pace of QE3 would not signal a withdrawal of monetary stimulus and that they could continue the program for quite some time at a lower level or even increase it again if needed.

Most economists do not expect a tapering of bond buying until later in the year, in part because of weak readings on inflation.

But Williams' remarks prompted a drop in stocks and a rebound in the dollar, with the greenback gaining further on Friday as investors prepared for a lessening of stimulus.

"It seems many Fed officials are becoming increasingly uncomfortable with the $85 billion per-month rut they find themselves in," said Dana Saporta, an economist at Credit Suisse in New York.

Still, Saporta does not expect the first adjustment in the purchases until September, although she would not rule out a move in June.

"I do think they are concerned the longer they maintain this $85 billion pace, the more exaggerated or adverse the market reaction will be when the time comes finally to make an adjustment," she said. "The more flexible and varied they are, the less exaggerated the market reactions may be."

Bernanke has sought to emphasize exactly this flexibility.

In March, he said it makes more sense to have a variable policy in which the flow of purchases responds "in a more continuous or sensitive way to changes in the outlook."

The Fed's policy-setting committee memorialized that approach in the statement they issued after their last meeting, on May 1, saying they were "prepared to increase or reduce" the purchases as labor-market or inflation forecasts change.

Bernanke is scheduled to hold a news conference after the June meeting where he could explain any policy shift and try to assuage anxious investors that the Fed remains highly accommodative. The next meeting is in late July, but the next opportunity to talk to the media is not until mid-September.

SOMETHING LESS THAN 'SUBSTANTIAL'

Since the Great Recession, the Fed has kept interest rates near zero and taken other extraordinary steps to get Americans back to work, including promising to keep buying bonds until the labor market outlook improves "substantially."

But it has never specified what it might take to prompt a reduction in its monthly purchases.

Monthly payroll growth has averaged 208,000 in the last six months, compared with 141,000 in the six months prior to the launch of QE3 in September. The unemployment rate dropped to 7.5 percent last month from 8.1 percent in August.

Policy hawks have seized on this improvement to argue that tapering should begin.

"I don't think there is any question ... that we've seen substantial improvement in the labor market outlook over the last six months," Jeffrey Lacker, president of the Federal Reserve Bank of Richmond, said earlier this month.

Williams, for his part, said he was more confident the upturn in the labor market would endure, saying that "nearly all" of the indicators he watches suggest further gains over the next six months.

One gauge of future economic growth compiled by the Economic Cycle Research Institute recently hit a two-year high.

STAY THE COURSE?

Still, the economic picture is far from clear and the recovery has stumbled midway in each of the last three years.

The housing market has continued its slow rebound, while retail sales unexpectedly jumped last month, prompting some economists to boost gross domestic product growth projections.

But the factory sector looks headed for a third straight monthly decline and most forecasters expect the economy to expand at only about a 2 percent annual rate in the second quarter. Growth of more than 2.5 percent over several quarters is generally needed to lower the still high jobless rate.

Inflation has also slowed, which is a key reason most Wall Street economists are betting the central bank stays the course for a few more months. Consumer prices rose just 1.1 percent in the year through April, well below the Fed's 2 percent goal.

Notably, however, the Fed's policy doves have not jumped on weaker inflation as reason to boost the monthly bond buys.

"I think it's way too early to think like that," Chicago Fed President Charles Evans said earlier this month.

'BALANCING ACT'

The risks of buying so many bonds might take center stage on Wednesday when Bernanke testifies to Congress and the central bank releases minutes of its most recent policy meeting.

Concerns have grown that the longer the Fed snaps up assets at the current pace, the greater the risk that bond and stock markets - which have run up in the QE3 period - will snap back violently when purchases finally slow.

"I think the Fed's got a very difficult balancing act here in removing ... QE from the market," Goldman Sachs President Gary Cohn told CNBC on Wednesday. "It wouldn't be surprising to me if they tried little maneuvers here and there."

Yields on the benchmark 10-year Treasury note are likely to rise only by about half a percentage point by the end of the year if the Fed begins to trim monthly bond purchases in December, as JPMorgan currently expects, economists at that bank said.

"The Fed is really trying to avoid surprises to the market," said Garth Friesen, a principal at hedge fund III Associates who also sits on the New York Fed's investor advisory committee.

"But as long as we don't get deterioration in the labor market, that could also tilt in the favor of discussing tapering in June, or having it start sooner rather than later."

(Reporting by Ann Saphir and Jonathan Spicer. Editing by Andre Grenon)