Most writers debate at least once in their careers if they should go the traditional route, self-publish, or something in-between.
And as an agent, I'm supposed to tell you you should always go traditional, right?
Wrong.
Although I have personally chosen the traditional route, and my clients, for the most part, have as well, the path you choose depends on your goal for the book.
If you want your book out fast, or to a more eclectic/niche market, maybe go for the Indie market.
If you want your book to have a wide-audience appeal, try for traditional.
When to Self-Publish
According to "Self-publishing: A Creative Solution to Academic Survival in the Commercial World," (Scholarly source) by Thomas Peter Stehlik, published by International Journal of the Book, self-publishing can be a good solution to academic authors who want to publicize findings but fail to break into the commercial fiction publishing industry.
Some other sources suggest the-self publishing route for the following reasons:
You will receive higher royalties ("Self-Publishing vs. Traditional Publishing," Zsofia Macho, Publish Drive)
You will have total control over the cover, content, and everything in-between ("Pros And Cons Of Traditional Publishing vs Self-Publishing," Joanna Pen, The Creative Pen)
It moves a lot faster than the traditional publishing route. ("Traditional Publishing versus Self-Publishing, no author listed, Scribendi)
Granted various cons accompany the self-publishing route as well. For instance, Indie authors have a hard time gaining reviews and the ability to sell their books in paperback stores. Not to mention, it requires a significant amount of funds and marketing expertise.
Some self-published authors choose to go the Vanity Press route to alleviate this extra marketing hat placed upon a writer. Some of these publishing houses can charge authors thousands of dollars.
When to Traditionally Publish
As a literary agent, here are the most common denominators I see when a publisher will pick up a client's book of mine.
The author has a significant platform.
This especially proves true in the non-fiction market. Publishers will not even look at memoirs unless the author has neared celebrity status.
The author has three or more comparative titles.
Granted, some books, such as Harry Potter, were unprecedented. However, traditional publishers now play a much safer game. They want to know if any similar books have sold well in the market.
The author has enormous patience.
Some publishers have a queue of agented submissions that stands at about a year. Others more. A book will often go 2-3 years from creation to publication in the traditional market, often longer.
The author is willing to go a more commercial route.
I hate formulas. Part of me wants to create something never introduced to the market before. However, authors who want to write traditional do have to stick to some rules, some tropes . . . some formulas. One of my writing instructors once said, "You have to know the rules before you can break them." The traditional market operates in the same way. Follow the rule, applaud the exception.
Some other resources: