You can not enhance the drop down list formatselect. But you can enhance with the hook tiny_mce_before_init the second drop down styleselect, there is hide in a default WordPress install. The documentation list all defaults and possibilities.

• inline – Name of the inline element to produce for example “span”. The current text selection will be wrapped in this inline element.
• block – Name of the block element to produce for example “h1″. Existing block elements within the selection gets replaced with the new block element.
• selector – CSS 3 selector pattern to find elements within the selection by. This can be used to apply classes to specific elements or complex things like odd rows in a table.
• classes – Space separated list of classes to apply the the selected elements or the new inline/block element.
• styles – Name/value object with CSS tyle items to apply such as color etc.
• attributes – Name/value object with attributes to apply to the selected elements or the new inline/block element.
• exact – Disables the merge similar styles feature when used. This is needed for some CSS inheritance issues such as text-decoration for underline/strikethough.
• wrapper – State that tells that the current format is a container format for block elements. For example a div wrapper or blockquote.

The Style Button

Note that, by default, the Style dropdown is hidden in WordPress. At first add the button for custom formats to a menu bar of TinyMCE, in example line 2 with hook mce_buttons_2.

add_filter( 'mce_buttons_2', 'fb_mce_editor_buttons' );
function fb_mce_editor_buttons( $buttons ) {

    array_unshift( $buttons, 'styleselect' );
    return $buttons;
}

The Custom Styles

Then enhance the drop down of this button. Aslo useful the enancement via additional value in the array, see the codex for this example.

/**
 * Add styles/classes to the "Styles" drop-down
 */ 
add_filter( 'tiny_mce_before_init', 'fb_mce_before_init' );

function fb_mce_before_init( $settings ) {

    $style_formats = array(
        array(
            'title' => 'Download Link',
            'selector' => 'a',
            'classes' => 'download'
            ),
        array(
            'title' => 'My Test',
            'selector' => 'p',
            'classes' => 'mytest',
        ),
        array(
            'title' => 'AlertBox',
            'block' => 'div',
            'classes' => 'alert_box',
            'wrapper' => true
        ),
        array(
            'title' => 'Red Uppercase Text',
            'inline' => 'span',
            'styles' => array(
                'color'         => 'red', // or hex value #ff0000
                'fontWeight'    => 'bold',
                'textTransform' => 'uppercase'
            )
        )
    );

    $settings['style_formats'] = json_encode( $style_formats );

    return $settings;

}

Result

Enhanced Drop down menu

You can also enhance the drop down with a tree menu. Create the var from the example source above with more structure in the array, like the follow source.

    $style_formats = array(
        array(
            'title' => 'Headers',
                'items' => array(
                array(
                    'title' => 'Header 1',
                    'format' => 'h1',
                    'icon' => 'bold'
                ),
                array(
                    'title' => 'Header 2',
                    'format' => 'h2',
                    'icon' => 'bold'
                )
            )
        ),
        array(
            'title' => 'Download Link',
            'selector' => 'a',
            'classes' => 'download'
        )
    );

For more possibilities and parameters see the default values of the Style Format Drop down field (write in javascript). var defaultStyleFormats = [ {title: ‘Headers’, items: [ {title: ‘Header 1′, format: ‘h1′}, {title: ‘Header 2′, format: ‘h2′}, {title: ‘Header 3′, format: ‘h3′}, {title: ‘Header 4′, format: ‘h4′}, {title: ‘Header 5′, format: ‘h5′}, {title: ‘Header 6′, format: ‘h6′} ]},

            {title: 'Inline', items: [
                {title: 'Bold', icon: 'bold', format: 'bold'},
                {title: 'Italic', icon: 'italic', format: 'italic'},
                {title: 'Underline', icon: 'underline', format: 'underline'},
                {title: 'Strikethrough', icon: 'strikethrough', format: 'strikethrough'},
                {title: 'Superscript', icon: 'superscript', format: 'superscript'},
                {title: 'Subscript', icon: 'subscript', format: 'subscript'},
                {title: 'Code', icon: 'code', format: 'code'}
            ]},

            {title: 'Blocks', items: [
                {title: 'Paragraph', format: 'p'},
                {title: 'Blockquote', format: 'blockquote'},
                {title: 'Div', format: 'div'},
                {title: 'Pre', format: 'pre'}
            ]},

            {title: 'Alignment', items: [
                {title: 'Left', icon: 'alignleft', format: 'alignleft'},
                {title: 'Center', icon: 'aligncenter', format: 'aligncenter'},
                {title: 'Right', icon: 'alignright', format: 'alignright'},
                {title: 'Justify', icon: 'alignjustify', format: 'alignjustify'}
            ]}
        ];

Add Custom Stylesheet to the editor

Now is the last point, that you include the custom css for this formats, via hook mce_css.

/**
 * Apply styles to the visual editor
 */  
add_filter( 'mce_css', 'fb_mcekit_editor_style');
function fb_mcekit_editor_style($url) {

    if ( ! empty( $url ) )
        $url .= ',';

    // Retrieves the plugin directory URL
    // Change the path here if using different directories
    $url .= trailingslashit( plugin_dir_url( __FILE__ ) ) . '/my-editor-styles.css';

    return $url;
}

Don’t forget to add this stylesheet rules also to the front end stylesheet.

Remove the Format Button

As enhancement you can remove the formatselect drop down button via check for the value in the button array. Add the follow source to the function fb_mce_editor_buttons at the hook mce_buttons_2.

$value = array_search( 'formatselect', $buttons );
if ( FALSE !== $value ) {
    foreach ( $buttons as $key => $value ) {
        if ( 'formatselect' === $value )
            unset( $buttons[$key] );
    }
}