It's also about personal preference and not doing more than you can keep up with as well. I agree with Tigger about not putting all of your eggs in one basket but all the eggs don't have to adult related either.
I run three other non-adult related businesses and a hobby blog. I used to do more but doing so became more than I could be successful at. Some people especially back when doing adult sites was very easy and profitable had many sites (100s) that were driving affiliate sales, etc. This could get lengthy but basically there are many flavors of the adult internet entrepreneur. You can mix and match or stick to what you know but the most important thing I learned is to be unique and flexible if nothing else.
The thought of running dozens or more blogs or sites was always a put off to me. I used to have tons of domains, what a nightmare. hahahahaha But it certainly does work for many people to keep up with that many sites. I just wanted to put in my 2cents.
I also have to say it wasn't until I started doing what I LOVED to do (for my adult and non-adult businesses) did I actually start to make money. So a good question to ask yourself is "How do I make money from what I love to do?" otherwise, it's also ok to sell out and make some frivolous loot too. hahahahahaha I do both.

... but I wouldn't accept only making money on things I really didn't love doing.