Supporters behind the goal, making a 250-mile mid-week round trip, celebrated wildly as Harvey’s 94th minute strike secured all three points rather than the share of the spoils that seemed on the cards.
Both Chichester and Bay fielded unchanged sides to the ones that beat Sittingbourne and East Grinstead respectively last time out.
If the visitors probably edged the first half, it was Chi who had the lion’s share of possession after the break.
A nice ping from left back Rob Hutchings towards top scorer Callum Overton earned the hosts an early throw-in and then Ethan Prichard flashed an effort narrowly wide of the post two minutes in.
Lewis Hyde gave the ball away uncharacteristically sixty seconds later but fellow defender Ryan Davidson got it back to keeper Kieran Magee.
Herne Bay pressed, as they would on several occasions, down the left flank and Ben Pashley had to have his wits about him to clear.
Next, Prichard lifted a pass towards Overton only for Bay defenders to turn the ball out.
Adem Ramadan and Antonio Hamilton combined but the final pass fed through to Zak Ansah found the leading marksman in an offside position.
Jack Parter then rattled Magee’s crossbar after a super ball from Ramadan and Pashley hooked it away.
The Chi No6 was involved moments later getting in ahead of Ansah following a raking pass.
Kaleem Haitham and Prichard connected in a slick move at the other end that resulted in a Ben Mendoza diving header which was just off target.
Mendoza had a cross cut out in the next attack and Prichard did well to break up a counter orchestrated by Hamilton, Dan Carrington and Ramadan.
Ansah, with 15 goals in all competitions this season, was flagged offside for a second time on 21 minutes before Prichard was impeded.
Mendoza rolled the free-kick short to Lloyd Rowlatt but a decent string of passes between Jamie Horncastle, Rowlatt, Haitham, Pashley, Hutchings and Overton eventually fizzled out.
Ramadan’s attempt was easy enough for Magee and the same Bay player hit a set-piece straight at the Chi keeper on the half hour mark.
Rowlatt then jinked into the area but for all his efforts the midfielder couldn’t get a shot off.
Magee gathered another low drive without any trouble and a cross-cum-shot from Prichard deflected off a defender for the home side’s first corner which came to nothing.
Lovely skill by Haitham forced Carrington into a foul and a booking before Bay broke once more only for the Chi back four to snuff out the danger.
Lewis Hyde caught Kieron Campbell but Magee dealt comfortably with Ramadan’s resulting free-kick.
Nice build-up play got Overton into the visitors’ penalty area but the referee wasn’t interested when the No9 went over.
Mendoza then whipped in a cross on the stroke of half time that was steered out for a corner which Hutchings swung in on to the roof of the net.
Emmett Dunn replaced Hyde at the break with Horncastle dropping back to partner Pashley.
Like Overton, Campbell went down in the box – nothing doing again according to the ref.
A mistake by Horncastle might have got Roberto Ratti in, but the Herne Bay signing from Ashford United, who scored twice in that five-goal-thriller at Oaklands Park in December, was denied this time by Rowlatt.
Chi’s back line held firm as Hamilton and then Carrington sought to get things going.
Haitham’s clever backheel sent Prichard away down the left before his cross came off an opponent’s boot and safely into the arms of Jordan Perrin.
Prichard tried to trick his marker who wasn’t fooled and the move broke down; Overton’s return pass to Davidson was too firm; and a key moment came and went when Harvey seemingly handled Mendoza’s shot and the ref dismissed vociferous penalty appeals.
City coped with a three-minute-or-so spell of pressure before the hour.
Dunn made a lovely tackle on Ramadan but Parter’s one on Haitham was cynical and earned him a yellow card.
Hutchings’ fizzed his free-kick inches away from in-coming teammates before a retribution challenge by Haitham on Parter saw his name go into the notepad as well.
Rowlatt, Mendoza and Haitham hooked up as the hosts probed but the final cross was diverted.
The referee waved away another claim for a stonewall Chi penalty with 20 to go and Overton was inches away from getting something on Mendoza’s delicious cross.
Magee then claimed a delivery from the right; fellow players bravely blocked Campbell’s effort; and Hutchings’ long-ranger rose over the frame of the goal.
A fierce shot from Rowlatt was too hot to handle for Perrin and the Bay custodian got away with one as his save of sorts spun out for a corner.
The visitors countered from this but Prichard did well to tidy up.
Bay rang the changes and Pashley headed clear a delivery from one of the subs Aaron Millbank.
Davidson then played a super defence splitting pass but Prichard was flagged offside before Haitham spurned a glorious chance firing into Perrin’s gloves.
Mendoza and Davidson broke down the right wing only for Davidson’s shot to swerve into the side-netting.
Ansah got away with controlling the ball with his hands, fed Bode Anidugebe who moved it on to substitute Rory Smith and Hutchings crunched him.
Horncastle got down low to block the subsequent effort, and how Parter didn’t score from close range in the 86th minute is anybody’s guess – Magee making a super save.
Rowlatt, Mendoza and Prichard got Chi going once more but the final ball to Overton was too strong.
Smith wasted another opportunity in time added on before Harvey broke Chi hearts with a decisive final touch after a goal-mouth scramble in the 94th minute.
The dramatic win sent Herne Bay up into fourth spot, level on points with third placed Ramsgate who have a better goal difference. Chichester remain ninth.
Chi welcome second placed Cray Valley PM to Oaklands Park on Saturday 12th February (Kick Off 3pm).
Chichester – Magee, Davidson, Hutchings, Horncastle, Hyde, Pashley, Rowlatt, Mendoza, Overton, Prichard, Haitham. Subs: (Carroll, Barry, Dellaud, Dunn, Farrier)