The company announced last week that its proposal to acquire the business, which is currently trading in Chapter 11, had "expired".
RR Donnelley initially made a bid of $1.35bn (£891m) for the business, which it is understood to have raised to $1.5bn following discussions with Quebecor's creditors.
Thomas Quinlan, president and chief executive of RR Donnelley, said that he was "disappointed" by the decision, adding that the proposal was "in the best interests of the creditors".
He added: "We look forward to continuing to pursue other strategic initiatives."
Quebecor will now continue with is proposal to emerge from bankruptcy protection this summer as a standalone business.
RR Donnelley drops bid for Quebecor World
RR Donnelley has dropped its bid for rival print giant Quebecor World, ending what would have been the biggest print merger in history.
