Bayesian statistical methods are based on the idea that one can assert prior
probability distributions for parameters of interest. Although this makes Bayesian analysis seem subjective, there are a number of advantages to Bayesianism. It tends to permit more direct conclusions about parameters than the frequentist approach and, once a prior is established, estimation and testing procedures tend to be straightforward. The major downsides of Bayesianism are the requirement that a prior be chosen and the tendency of Bayesian methods to require a great deal of computational power. However, if you do have a lot of prior information, it can be extremely beneficial to incorporate that information into an estimation procedure. We try to find books that offer the Bayesian perspective for all the statistical topics on this site, but most applied books are not strictly Bayesian. However, the books in this category give the orthodox Bayesian perspective.
Recommended Books

A First Course in Bayesian Statistical Methods
Peter D. Hoff
Key Features
 Intext exercises
 Errata and R code
Key Topics
 Belief
 Binomial Model
 Conjugate Priors
 Exchangeability
 Gibbs Sampling
 Group Comparisons
 Hierarchical Modeling
 Latent Variable Methods
 Linear Regression
 MetropolisHastings Algorithms
 Mixed Effects Models
 Monte Carlo Approximation
 Multivariate Gaussian/Normal Model
 Poisson Model
 Probability
 Univariate Gaussian/Normal Model
Description
This is an excellent and concise introduction to Bayesian techniques. It will take you all the way from simple, onedimensional, conjugatepriorbased methods to probit regression. There is a (necessary) focus on Monte Carlo techniques throughout most of the book. There are a limited number of exercises, but the ones that have been included are effective. We didn't realize it when we first read this book, but it is extremely important to note that Hoff has provided the R code he uses at the website linked above. Reading that code can be very helpful for the application exercises.

Bayesian Data Analysis
Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin
Key Features
 Intext exercises
 Errata etc.
Key Topics
 Accounting for Data Collection
 Assessing MCMC Convergence
 Asymptotics
 Bayesian Computation
 Bayesian Inference
 Bayesian Interpretation of NonBayesian Approaches
 Comparing Models
 Computationally Efficient Simulation
 Decision Analysis
 Dirichlet Process Models
 Finite Mixture Models
 Gaussian Process Models
 Generalized Linear Models
 Gibbs Sampling
 Hierarchical Linear Models
 Hierarchical Models
 Importance Sampling
 Informative vs Noninformative Priors
 Metropolis Hastings Algorithms
 Modal and Distributional Approximations
 Model Checking
 Models for Missing Data
 Models for Robust Inference
 Multiparameter Models
 Multiple Imputation
 Nonlinear Models
 Nonparametric Models
 Nuisance Parameters
 Numerical Integration
 Probability and Inference
 Regression Models
 Single Parameter Models
 Variational Inference
Description
If you've decided you want to convert to Bayesianism, you ought to read the gospel according to Gelman. We like this book; reading it is like having a long conversation with someone with a lot of experience. However, we often found ourselves having to turn to outside resources in order to complete the exercises, which we attribute to a lack of code examples and somewhat summary, albeit verbose, exposition. You'll learn a lot about practical Bayesian analysis by reading this book; just don't expect it to be selfcontained.