We raise dairy steers from birth to finish and get a lot of practice with starting calves.  As I stated earlier, what you have described sounds a lot like what my wife calls "stupid tongue."  Our experience shows that the number one cause of "stupid tongue" is a sour stomach from overfeeding.  The calf is full and is not hungry and forcing extra milk into the calf causes the milk to sour in the stomach because some of the milk lingers too long since the stomach never fully empties.

Generally with an 80-120# dairy calf you can get away with 2qt 2X daily and not have issues with slow digestion.  Try and follow the same routine with a 40# Jersey you start having issues from over feeding since calf is so small; so guys often switch to 3-4x daily feedings for the first few weeks.  Try 2qt 2x daily it with a 60# Angus calf and you get issues to since they have been indirectly genetically selected for more smaller feedings since most Angus calves nurse at will, not 2X daily when the farmer wants to feed them.  Unless it is extremely cold (-10F or lower) we don't get too worried about a calf skipping a feeding or 1.5qt 2X daily for a few days.

Since you left the calf with the cow there is a very good chance that it nursed while you were not watching.  Now if you go and tube it because it won't drink and overfill the stomach for a few days, eventually the stomach gets sour and then refuses to drink; I know when I have an upset stomach hitting up an all you can eat buffet is no where near the top of the list of things to do.  Most any bottle calf raisers will tell you there is nothing wrong with a hungry calf come feeding time, and you have less issues with slightly hungry calves than you do with fully pacified calves since often when they are not hungry they will wait an hour or two until they are hungry before consuming their milk which makes the milk the wrong temperature and causes clostridia flareups.

I also should have asked earlier: What is the color and consistency of the stool?  Are you weighing your milk replacer powder?  Mixing according to directions of manufacturer?  What temp is the milk when you are feeding it?