Calculus: Early Transcendentals, Chapter 7, 7.2, Section 7.2, Problem 28

inttan^5(x)sec^3(x)dx
Rewrite the integrand as,
=inttan^4(x)tanx(x)sec^3(x)dx
Now use the identity: tan^2(x)=sec^2(x)-1
=int(sec^2(x)-1)^2sec^3(x)tan(x)dx
Now apply the integral substitution,
Let u=sec(x)
du=sec(x)tan(x)dx
=int(u^2-1)^2u^2du
=int(u^4-2u^2+1)u^2u
=int(u^6-2u^4+u^2)du
=intu^6du-2intu^4du+intu^2du
=u^7/7-2(u^5/5)+u^3/3
Substitute back u=sec(x)
=1/7sec^7(x)-2/5sec^5(x)+1/3sec^3(x)
Add a constant C to the solution,
=1/7sec^7(x)-2/5sec^5(x)+1/3sec^3(x)+C