India, without captain and leading batsman Shubman Gill who was ruled out because of a neck injury he suffered on Saturday, faltered in their chase of 124 to be bowled out for 93 in Kolkata.
Harmer, who had figures of 4-30 in the first Indian innings, struck key blows in the second session including getting Rishabh Pant caught and bowled for two to dent the opposition chase.
Left-arm spinner Keshav Maharaj struck twice in two balls to pack off India and trigger wild celebrations in the South African camp with the reigning world Test champions taking a 1-0 lead in the two-match series.
Washington Sundar attempted to anchor the chase but fell to Aiden Markram's off-spin for 31.
Axar Patel gave the crowd some cheer with his two sixes off Maharaj before the bowler had his revenge to send the batter caught out for 26 off 17 balls.
Pace spearhead Marco Jansen ripped out the Indian openers before lunch with the hosts effectively 10-3 in the absence of Gill, who is in hospital and "under observation", at the break and Harmer soon took charge.
Jansen in his first over got the left-handed Yashasvi Jaiswal to nick a length delivery to wicketkeeper Kyle Verreynne, who then took a diving catch to dismiss Rahul.
South Africa skipper Temba Bavuma hit a defiant 55 not out in his team's second innings total of 153.
Bavuma played a defiant knock to keep South Africa in the hunt after they resumed on 93-7 on a tough pitch with inconsistent bounce.
He started the day on 29 and played with grit to complete the first half-century of a low-scoring contest.
Bavuma's 44-run eighth-wicket stand with overnight partner Corbin Bosch frustrated the Indian bowlers until pace spearhead Jasprit Bumrah broke through.
Bumrah bowled Bosch for 25 before Bavuma reached his fifty in 122 balls, raising his bat to an applauding dressing room.
Fast bowler Mohammed Siraj shattered Harmer's stumps for seven and removed last man Keshav Maharaj for a duck in the space of four balls as Bavuma ran out of partners.
Left-arm spinner Ravindra Jadeja had figures 4-50.
South Africa fought back on day two after being all out for 159 on Friday after winning the toss and electing to bat.
They dismissed India for 189 on Saturday to keep the deficit to 30 before again faltering in their batting.
The second Test starts Saturday in Guwahati.
