Calculus of a Single Variable, Chapter 8, 8.8, Section 8.8, Problem 23

We will use integration by parts
int u dv=uv-int v du
We will need to apply integration by parts two times in order to eliminate x^2 from under the integral.
int_0^infty x^2e^-x dx=|[u=x^2,dv=e^-x dx],[du=2x dx, v=-e^-x]|=
-x^2e^-x+2int_0^infty xe^-x dx=|[u=x,dv=e^-x dx],[du=dx,v=-e^-x]|=
(-x^2e^-x-2xe^-x)|_0^infty+2int_0^infty e^-x dx=
(-x^2e^-x-2xe^-x-2e^-x)|_0^infty=
lim_(x to infty)(-x^2e^-x-2xe^-x-2e^-x)-(-0-0-2)=
To calculate the above limit we will use L'Hospital's rule:
lim_(x to a) (f(x))/(g(x))=lim_(x to a)(f'(x))/(f'(x))
First we rewrite the limit to fit the form of L'Hospital's rule.
lim_(x to infty) -x^2e^-x=lim_(x to infty)-x^2/e^x=
Now we differentiate.
lim_(x to infty)-(2x)/e^x=
This integral still yields infty/infty so we differentiate again. Notice that this corresponds to the second term in our calculation of the integral.
lim_(x to infty)-2/e^x=0
Now we know everything needed to calculate the integral.
-0-0-0+0+0+2=2
As we can see the integral converges to 2.
The image below shows the graph of the function and area under it corresponding to the integral. We can see that the function converges to zero "very fast" (there's hardly any area under the graph after 12). This implies likely convergence of the integral.