The Borg have superior numbers and technology the Dominion would not have stood a chance. The Jem'Hadar who form the backbone of the Dominion military machine would have been easily assimilated, in addition their dependence on ketracel white would have been a weakness the Borg could effortlessly exploit.
DS9:In The Pale Moonlight showcased that the toehold the Dominion established in the Alpha Quadrant, cut off from massive reinforcements they could have summoned from the Gamma Quadrant, could lay waste to two out of three major powers. This is because Jem'Hadar breed like fruit flies, and Dominion shipbuilding is efficient.
The Borg may be even more efficient when it comes to building ships, but when it comes to manpower, they've only got assilimation, and it leaves a lot to be desired. Unlike the Federation, the Dominion isn't going to wring its hands over killing nanoprobe-infected crew, or laying waste to innocent planets just so the Borg don't swing by for fresh meat.
The Borg might have a technical edge, but since the "technology" is where the show tends to fudge things the most, your mileage may vary. You might be thinking about the Voyager-style Borg that keeps technology around for the sole purpose of allowing enemy strike teams to steal it with the help of a simple distraction... or you might be thinking about the TNG Borg that have impenetrable shields and unblockable weapons.
But really, the Borg and the Dominion are natural allies. They both love rigidly imposed order. And since the Borg are more predictable than the rabble of various other species that the Dominion has to keep a tight lid on, I wouldn't be suprised if the Founders let the Borg have their way with the Solids in the Gamma Quadrant.
We've never seen evidence of the Borg allying themselves with anyone (except Janeway, and that partnership was pretense). No one in the Gamma Quadrant has more biological distintiveness to catch the Borg's interest than the Founders. I don't think the Jem'Hadar would really pose much of a challenge to the Borg, and once they are out of the picture attempts would be made to assimilate the Great Link -- probably not in the same manner they assimilate humanoids, but they'd figure something out.
You can't be serious about the Dominion winning. Think back to Wolf 359 one Borg ship destroyed 39 Federation starships. The only way Starfleet was ever able to beat the Borg was to outthink them but that won't happen with the Dominion. If you have a ship full of Jem'Hadar all you have is a ship of fighters no scientists or engineers to outthink the collective. Not only that but all the Borg need to do is assimilate a single Vorta or Jem'Hadar and they have instant knowledge of all weaknesses and battle strategies.
If that the fact that the Dominion is willing to slaughter its own troops if they think they may have been infected with Borg nanoprobes is going to give them some advantage you forget that the Dominion troops need daily infusions of ketracel white to function. In the Dominion War the fact that the Dominion couldn't keep up its ketracel white production to the same level as it was conquering systems meant that it had to stop advancing. Just think if the Borg managed to find a way to infect their entire supply of the drug, and as stated above if one cube took down 39 Federation ships I don't Jem'Hadar fighters would do any better.
Dominion vs. Borg= Winner Borg!
Not an easy question to answer, really. Most people will favor the Borg in this scenario, of course. Best of Both Worlds really sticks out in people's minds when it comes to considering how devastating and relentless the Borg can be. I'm not discounting how tough the fight would be, but to play devil's advocate let's discuss a few things.
1. There is a limit to what the Borg can adapt to. You can see this in Species 8472. Their weapons were powerful enough to consistently destroy the Borg in every confrontation. In the similar vein, the Borg can only get so close to a Sun without being damaged and crash landings have destroyed whole spheres and cubes. Hence the conclusion: a large enough energy discharge or an object with enough mass or speed CAN destroy a Borg cube outright, and the Borg cannot adapt enough to prevent that destruction.
2. As per Star Trek: First Contact the Borg are vulnerable to hand-to-hand combat and possibly kinetic weapons such as guns.
3. Viruses and indiviuality can destroy whole Borg cubes.
4. The Dominion is stronger than the Federation. They have better technology, better manpower, etc. So all the tricks the Feddies have pulled to deal with the Borg? The Dominion can find those same solutions, quite possibly with greater speed and efficiency than their Alpha Quadrant counterparts. And they most certainly will not have any qualms about, for example, inducing a fatal virus into a captured Borg soldier so as to destroy a whole ship or a fleet or a planet.
5. The Dominion compartmentalize. The Founders literally breed their loyal citizens to do one job, and breed them to do that job extremely well. Vorta are the diplomats and the strategists, Jem-Hadar are the warriors, etc, etc. This will mean that, while not impossible for the Borg to obtain a "assimilate Picard" moment as they did with the Feddies (e.g. use one person's knowledge to adapt completely to a civilization's technology and weapons) it will be HARDER. Assimilate a Jem'Hardar, you'll get an idea about tactics, but not the minute details about a ship's weapons.
6. The Dominion are fanatics. When we first meet the Jem'Hadar, they plow their ship right into an Enterprise model vessel for no other reason than to send a message- we are willing to die in the service of our Founders. This means two things. First, Dominion citizens will take their own lives rather than be assimilated. Second, the Dominion are willing to use kamikaze strategies to kill their enemies. In Best of Both Worlds Riker is prepared to ram a Borg cube at maximum warp to stop them from assimilating Earth. Keeping point 1 in mind, do you think the Borg to adapt to such a maneuver? What if several ships did it at once? The Dominion will happily fight a war of attrition- give them a week and they can regrow all the soldiers they lose and rebuild all the ships they need.
7. Jem'Hadar are easily the deadliest warriors we've seen this side of the Klingon homeworld. They carry long range and melee weapons and train methodically in using both. They are the ultimate shock troops, each an expert in military strategy, and all bred and trained to fight with Borg like precision within their individual units. They are going to make assimilation painful and likely force the Borg to lose more drones than they gain.