Potter’s troubled side crashed to a seventh defeat in their last 10 games in all competitions.
Chelsea fell behind to a goal from their former winger Willian in the first half at Craven Cottage.
Kalidou Koulibaly equalised after the break but Felix was dismissed for a reckless studs-up lunge on Kenny Tete before Carlos Vinicius headed the late winner after Kepa’s mistake.
Joao Felix, who only joined Chelsea on Wednesday in a £9.7m loan deal until the end of the season from Atletico Madrid, was shown a straight red card for a horrible lunge on defender Kenny Tete after 58 minutes.
It was 1-1 at the time, with Willian putting the hosts ahead against his former side in the first half, before Kalidou Koulibaly equalised early in the second.
But the 10 men of Chelsea could not hang on to a point as Carlos Vinicius headed in a 72nd-minute winner.
Potter’s Blues side started brightly, but the hosts nearly took the lead when Chelsea lost possession and Carlos Vinicius played a pass through to Bobby de Cordova-Reid, who fired against the crossbar with Kepa Arrizabalaga beaten.
But Chelsea fell behind two minutes later in the 25th minute.
It was poor defending from the visitors as teenage left-back Lewis Hall failed to clear, before Trevoh Chalobah should have done better with his clearance.
The ball fell to Willian and the Brazil midfielder, a two-time Premier League winner during a seven-year spell at Chelsea, saw his strike take a big deflection off Chalobah to give Arrizabalaga no chance. Willian did not celebrate against his former team.
But Chelsea equalised within two minutes of the second half as Koulibaly reacted quickest to force the ball over the line from close range after Mason Mount’s free-kick had bounced off the left-hand post.
Joao Felix had been the visitors’ best player before his dismissal left his side on the back foot, with Vinicius scoring the winner.
Chelsea have now only won twice in 10 matches in all competitions.