"But now, what is the difference between Bayesian and Frequentist inference?", the answer is that the frequentist (as I understood it) conditions on theta, while the Bayesian averages over theta. That can be a big difference in settings where theta is high dimensional or where there is otherwise great uncertainty in theta. -- Andrew Gelman

Prediction

Given $X = \{x_1, \ldots, x_n\}$ i.i.d. observations, predict a new i.i.d. data point $\tilde{x}$.

Frequentist Context

Since $\theta$ is an unknown fixed value, it is not a random variable and is something we have to estimate. Therefore, we compute an (point/interval) estimator for the model, given the observed data. Then, we plug it in to the model and predict the new data.

1. Find $\theta$ by some means of estimation.
2. Using the estimated $\hat{\theta}$, plug in the $\hat{\theta}$ to the model.
3. Predict $\tilde{x}$ given the model.

Bayesian Context

Since $\theta$ is a random variable, we can compute the posterior predictive distribution.

$$p(\tilde{x}|X) = \int p(\tilde{x}, \theta|X) d\theta = \int p(\tilde{x}|\theta, X)p(\theta|X)d\theta$$

This is equivalent to computing the expected value of the distribution of new data over the posterior distribution.