I think it is currently not possible to do so. Setting the fontsize sets the font size for everything except the legend labels.