Do it. I have one of the cleanest LJ's (2006 LJ Rubicon) I could find and I sometimes think that if I had it to do over, I would have bought a little bit older and built a Brute.
Right now you have a great looking TJ and it's even better with the highline kit (I too have the kit), but how many times a day are you driving around and see another built TJ (maybe not quite as nice), but still a regular TJ??
Now think about how many Brutes you have EVER seen driving around. I think that's the part you are looking for. You mentioned that "hot rod" component and I think the Brute will give it to you in a way that nothing else that you could do to YOUR jeep will.
If you have the desire, means and ability, by all means go for it!!
2006 LJ Rubicon