Assuming that a chart of progress somewhat resembles a map, you can say that progress has gone down in the same way a route can go south... I did no research to answer this question but it makes absolute sense to me.

The expression "went south" is used to define failure, loss or a bad experience and started after the Civil War. "Went South" was used in a derogatory manner to infer that anything from the South was bad, a failure. The term is attributed to General William Sherman during the burning of Atlanta.

Everything went south: From the convention of having South be on the bottom of maps. To go south is to go downward, including circumstances, relationships, and stock prices. http://www.phrases.org.uk/bulletin_board/55/messages/577.html If a project has gone south, it's "in the toilet," "gone bust". into a state of decline or ruin. Causes the sluggish economy to go south -- G. F. Will. go south (also head south, take a turn south) 1 v phr by 1940s To disappear; fal by or as if by vanishing.... 2 v phr by 1925 To abscond with money, loot, etc. ... 3 v phr underworld by 1950 To cheat, esp to cheat at cards.... 4 v phr by 1980s To lessen; diminish.... [probably fr the notion of disappearing _south of the border_, to escape legal pursuit and responsibility; probably reinforced by the widespread belief that the soul after death journeys to the south, attested in American Colonial writing fr the middle 1770s; _GTT_, "Gone to Texas, absconded," is found by 1839] From _Dictionary of American Slang, Third Edition_ (1995) by Robert L. Chapman I've read that this is due to the downward slope of a badly performing stock or profit return as shown on a performance graph. The downward slope is reckoned to be similar to the southerly direction on a map, ie at the bottom. It doesn't represent any supposed worse aspect of the world 'down south', just the direction. North American pilots also have a variation on this expression --- it's 'gone west' When we have a meeting of pilots we will often start the meeting with a toast to those of us who have "gone west". We face the west and drink to those who have died. West, in this case, refers to the place the sun sets -- extinguishes -- the metaphor is clear. http://www.phrases.org.uk/bulletin_board/26/messages/658.html

from up north, where the smart people live.

Sources with NASA and JPL said they originated it during early rocket testing at White Sands Missile Range. Rockets were fired towards the bombing range towards the north. Rockets that malfunctioned often headed south instead, towards Mexico and had to be destroyed in flight. Hence the term for a malfunction coined "went south".

I think it might have something to do with either of these possibilities. It could be a Civil war reference, for the confederates (southerners) were a bit ornery back in the day, and even a little bit now as evidenced by confederate flags, though it has nothing to do with racism or slavery. The second possibility would be in reference to lewd conversations. A conversation could be said to go south when it goes beneath the belt or waistline, and while it could mean a mere low blow, it could mean potentially objectional humor.