An examination of the quest for Atonal music
There have been several periods in history when musicologists (and composers) have looked back at the way music is composed and attempted to codify it. In the 14th century Philipe de Vitry examined music of the day and coined the terms Ars Antiqua and Ars Nova (old music and new music). At the time there was a sense of music "breaking free" from the old re-strictures and creating something new - isorhythms and polyphony. Later, in Mozart's time (18th century), Jean-Philippe Rameau wrote the Treatise on Harmony which analyzed music of the day creating a series of fundamental laws governing Western music.
While many of Rameau's theory techniques are still in practice today when analyzing tonal music (the use of Roman numerals for chord symbols and the avoidance of parallel 5ths and octaves to name a few), the basic theory was an an examination of music of the period discovering how music was written, and not so much about how music should be written. It does put forth how music should be written, but only in so much as this is how good music currently is being written (current for Rameau's time) so if you wanted to write good music of the day it was sort of a how-to book. Vitry's book was more focused on how-to, creating a set of rules on which future composers could look for guidance. However, even that didn't really remain in effect as composers continued to stretch and bend the rules as they saw fit in a quest to write "good" music.
The point of both of these works were to encourage beautiful music. However, with the advent of the 20th century, and a pair of world wars, some composers have decided the world is not always a beautiful place and therefore music shouldn't always be beautiful. Ok... I can accept that premise.
The result, however, was a push to create a new set of rules for writing music. However, rather than analyze what makes "good" music, some of these rules ignored that concept and pushed for mathematical sense rather than artistic sense. The 12-tone system and later serial music both looked at composed music with rules and not aesthetics. The result was angular, atonal sounds that were anything but pleasing. If the goal was ugly music, they were successful.
The problem with this approach is it seems to me to be backwards, making the rules for composing music before the music is written, rather than the other way around. It would be rather like not liking fiction, so completely throwing out grammar and spelling rules to write new novels. This wouldn't work because the point of the grammar and spelling is to allow language to communicate. Without the rules (which, BTW, the grammar and spelling rules were written after language was in use, not before - much like the first two musical treaties I spoke of), language fails to have recognizable structure and communicates nothing.
Therefore, writing music to rules the audience does not understand or is unfamiliar with is counter-productive. This isn't to say the rules we have shouldn't be bent or broken or mangled in an attempt to create a new sound (and thus creating a new set of rules). However, it does mean writing music should be done from the understanding that these are the current set of rules; this is what people understand as music. Then, when you break these rules you do so with the knowledge that the audience will understand which rules your breaking.
ALSO, there should be some concept of making "good" music, not just mathematical sense. While music is mathematical, it is not just math. It is also art and art needs to have that emotional element.
Note: Some of the composers who attempted to create something new through 12-tone and serial music have (and are) created(ing) "good" music, music which future generations of composers will look at and examine for generating new rules of their own for which to compose by. However, I do not think it is either the 12-tone or serial methods they employed but rather something else they incorporated that makes their music good.