For the 2002 season, Spencer would join Chip Ganassi Racing and drive the #41 Target Dodge Intrepid. He began the season by failing to qualify for the Daytona 500, then had a streak of four top-five qualifying efforts, including at Bristol Motor Speedway, where he started fourth and was leading the race when he was bumped by Kurt Busch to win, starting a long rivalry between the two. After another DNQ at Watkins Glen International, Spencer was released from the ride at the end of the season, causing him to file a lawsuit against the Ganassi organization, saying his dismissal was a violation of his contract. During the season, he also won his most recent Busch Series race to date at Bristol driving for James Finch.
Spencer joined Ultra Motorsports in 2003, piloting the #7 Sirius Satellite Radio Dodge. He had four top-tens during the season, but at the running of the GFS Marketplace 400, Spencer's tires were nearly cut by Kurt Busch (who admitted it over the team's radio). After Busch intentionally stalled in front of Spencer's garage and reved his engine as a threat to block Spencer, the latter confronted Busch and punched him in the face. Spencer and Busch were both fined and placed on probation for the rest of the year, and while Spencer was suspended for one race, the incident led Busch to be heavily criticized by fans. Spencer ended the season 29th in points. He was also hired to drive three races for in the #2 Team ASE Racing Dodge Ram for Ultra's Truck Series team, winning the pole and the race in his second start at New Hampshire International Speedway. He became a part-owner of the Cicci organization that season, when he put Stuart Kirby in Cicci's #34 United States Air Force Chevy, but that partnership soon dissolved. He continued to remain involved as a part-owner, when he leased his shop to Bang! Racing in the Craftsman Truck Series in 2004.
He began 2004 with Ultra's Cup team at the Daytona 500, but when the team closed down due to a lack of sponsorship, he replaced Kevin Lepage at Morgan-McClure Motorsports, which had also been running unsponsored. Spencer's best finish that season had been 13th, when on October 25, he was arrested after trying to interfere with the police, who had a warrant to arrest his son for vandalism. The incident cost Spencer his job at Morgan-McClure, and he sat out the rest of 2004.
Spencer returned to the #2 Ultra truck in 2005. While he failed to win a race, he had nine top-ten finishes and finished twelfth in points. He did come close to a victory, however, in the season opener at Daytona. He held the lead late in the race and held off 2004 series champion Bobby Hamilton until just before the caution came out on the last lap. Assuming he was in the lead when NASCAR froze the field, he completed the caution lap then pulled into victory lane, only to be told by an official that, in fact, he had finished second. He also ran part-time in Cup, running nine races in the #50 Arnold Motorsports Dodge, and one race apiece for Peak Fitness Racing and R&J Racing.
When Arnold was unable to locate a sponsor and Ultra closed its doors following a fallout with the Ford Motor Company, Spencer began working full-time on SPEED. He had run both Cup races at Pocono Raceway for Furniture Row Racing in 2006, finishing 32nd and 36th, respectively. Spencer now works full time as an analyst for SPEED TV.
No speeding in the Park. Bicycles are allowed on all two tracked trails. Do not bike on single tracked trails. Wheels yield to feet, (Hikers and horses). Hikers yield to horses.
Cache placement approved by Martin Morse, Park Service Specialist. Obey all speed limits. Do not block gates. All wheels off the pavement.