House Speaker Nancy Pelosi announced in a letter to colleagues on Wednesday that Democrats will add paid family and medical leave back into their large social spending bill, but soon after, Democratic Sen. Joe Manchin called the move a “challenge.”

Democrats had initially called for 12 weeks of leave but it was cut down to four before being dropped altogether last week after Sen, Manchin, D-W.Va., raised concerns.

Now, Pelosi is saying it’s back in the bill with a source familiar confirming to ABC News it will be four weeks of paid family and medical leave.

However, from the Senate side, a source told ABC News the addition of paid family and medical leave is “far from reality yet” — and Pelosi acknowledges obstacles ahead in the upper chamber in her “Dear Colleague” letter.

“Because I have been informed by a Senator of opposition to a few of the priorities contained in our bill and because we must have legislation agreed to by the House and the Senate in the final version of the Build Back Better Act that we will send to the President’s desk, we must strive to find common ground in the legislation,” she wrote.

Walking into the Capitol Wednesday morning, Pelosi told reporters that she is “very sad” over Democrats’ loss in the Virginia governor’s race but said his loss does not change Democrats’ agenda in the House and is not the reason for the push to bring back the paid leave measure.

Manchin, meanwhile, did not explicitly say he would vote against the package if paid leave is included. But when asked about it shortly after word surfaced on Wednesday, he said he remains strongly opposed to it being included in the reconciliation package — a process Senate Democrats are using to avoid a Republican filibuster, but a tactic that requires all 50 Democratic votes, including Manchin’s. He told reporters he was unaware of the newly announced plans to include it in the package, but that House colleagues were familiar with his position on it.

“I just think it’s the wrong place to put it because it is a social expansion,” Manchin told reporters, repeating his concerns about the cost. He worries about “getting more debt and basically putting more social programs that we can’t pay for that we’re having problems with now.”

Rep. Richard Neal, D-Mass., chairman of the House Ways and Means Committee, said in a statement that paid family and medical leave will be fully paid for, and it is means-tested.

“We do this responsibly, fully paying for the means-tested program,” he wrote in a statement on Wednesday.

Sen. Kristen Gillibrand, D-N.Y., the lead sponsor of the family paid leave effort in the Senate, told ABC News she’s “very excited it’s back in the bill” and that she plans to speak to Manchin on it.

“Speaker Pelosi, I think, has decided it’s essential for the country and for workers to be able to rebound, and I trust the Speaker. I trust her judgment,” Gillibrand said.

Manchin, however, has for days said he’s concerned about insolvency in Medicare and Medicaid, and on Monday he said social expansion beyond those programs, for things like paid leave, is “aspirational”.

While he ultimately said he supports paid leave, he said he wants to see it shepherded through in a separate bill, not in reconciliation.

“We’re trying to force it through reconciliation which has guardrails and rules and regulations,” Manchin said. Let’s do it and do it right and not do it in this.”

Progressives have said they trust President Joe Biden to deliver 51 Senate votes he promised on the larger social spending and climate bill, and Biden predicted Manchin would support that in a speech on Tuesday, despite Manchin expressing new concerns.

Manchin said on Wednesday that the House will ultimately “do what the House does” and then the Senate will need to work through the bill “deliberately.”

Pelosi said updated bill text for the social spending plan will be released later Wednesday, and she anticipates the House Rules Committee will meet later in the day to debate the legislation and prepare it for votes on the House floor.

In a subtle message to House moderates, who are demanding at least 72 hours to read the bill text, Pelosi also noted that text was initially released six days ago, meaning they’ve had plenty of time to read up.

Based on Pelosi’s letter, it’s clear Democratic leadership intends to vote on both the already Senate-passed bipartisan infrastructure bill and the social spending bill Democrats plan to pass through reconciliation this week.

ABC News’ Trish Turner contributed to this report.