Lightly grease a 9x9 baking pan and line with parchment paper. Set aside.
Add condensed milk, sugar, and butter to a heavy saucepan. Place over medium heat and stir immediately.
Bring mixture to a gentle boil. Stir constantly for 12–15 minutes until thick and deep caramel brown.
Remove from heat. Stir in vanilla, coconut, peanuts, raisins, and spice if using.
Drop a small spoon into cold water. If it firms up quickly, the fudge is ready.
Pour mixture into prepared pan. Spread quickly and evenly.
Let cool for 20 minutes. Score into squares while warm. Allow to fully set before serving.