More accurate prognosis of hormone-positive early stage breast cancer patients offers the opportunity to make more informed follow-up choices, for example the addition of adjuvant chemotherapy. Traditionally, pathologists have prognosticated these cancers using conventional staging, tumor proliferation index, and a small set of morphological features manually scored from H&E slides. This information may be combined with the immunohistochemical (IHC) protein expression of the tumor. The rich information in these slides is summarized in terms of a simple univariate score such as the proportion of positively staining tumor nuclei. To investigate whether there is additional prognostic information in both the H&E and IHC slides, we constructed a prognostic model to predict recurrence risk from an exhaustive set of automatically calculated image features. On our whole slide cohort, the image-feature based recurrence risk binary classifier outperforms using clinical and expression covariates alone. Prognostic features include nuclei size, nuclear atypia, co-expression of ER and Ki67, lymphocyte density and stromal features. Our machine-learning based approach is a viable way to discover and integrate information holistically from different clinical prognostic data sources including clinical/demographic, H&E slide-based features and, for the first time, IHC stained slides.