Solution to Abstract Algebra by Dummit & Foote 3rd edition Chapter 2.1 Exercise 2.1.16
Let $n \in \mathbb{Z}^+$ and let $F$ be a field. Prove that the set $$UT_n(F) = \{ [a_{i,j}]_{i,j = 1}^n \in GL_n(F) \ |\ a_{i,j} = 0\ \mathrm{if}\ i > j \}$$ is a subgroup of $GL_n(F)$, called the group of upper triangular matrices.
Solution: First we prove some technical lemmas.
Lemma 1. If $A \in GL_n(F)$ is upper triangular, then no diagonal entry of $A$ is 0.
Proof: The determinant of a matrix can be computed using $$\mathsf{det}(A) = \sum_{\sigma \in S_n} \mathsf{sgn}(\sigma)\prod_{i=1}^n a_{i,\sigma(i)}.$$ But since $a_{i,j} = 0$ for all $i > j$, any permutation which carries an integer to a smaller integer contributes 0 to the sum. The only permutation not having this property is the identity, so that $\mathsf{det}(A) = \prod_{i=1}^n a_{i,i}$. Now since $\mathsf{det}(A) \neq 0$, no diagonal entry is zero.
Lemma 2. If $A, B \in UT_n(F)$, then $AB \in UT_n(F)$.
Proof: Say $A = [a_{i,j}]_{i,j = 1}^n$ and $B = [b_{i,j}]_{i,j = 1}^n$. Then $$AB = [\sum_{k=1}^n a_{i,k} b_{k,j}]_{i,j=1}^n.$$ Now suppose $i > j$. If $k>j$, then $b_{k,j} = 0$. If $k\leq j$, then $a_{i,k}=0$. Thus the sum is zero, and so $AB$ is upper triangular.
Lemma 3. If $AB = I$ and $A$ is upper triangular, then $B$ is upper triangular.
Proof: Say $A = [a_{i,j}]_{i,j = 1}^n$ and $B = [b_{i,j}]_{i,j = 1}^n$; then we have $$[\sum_{k=1}^n a_{i,k} b_{k,j}]_{i,j=1}^n = [\delta_{i,j}]_{i,j=1}^n,$$ where $\delta_{i,j} = 1$ if $i = j$ and 0 otherwise. We show that $b_{i,j} = 0$ for all $i > j$ by induction on $i$. For the base case, set $i = n$ and let $j < i$. Then $$\sum_{k=1}^n a_{n,k} b_{k,j} = a_{n,n} b_{n,j} = 0.$$ By Lemma 1, we know that $a_{n,n} \neq 0$, so that $b_{n,j} = 0$. Now for the inductive step, let $1 \leq i < n$ and $j < i$ and $m < \ell, b_{\ell,m} = 0$. Consider $\sum_{k=1}^n a_{i,k} b_{k,j}$; If $k > i$, then $b_{k,j} = 0$ by the induction hypothesis. Then the sum is $a_{i,i} b_{i,j} = 0$; by Lemma 1, $a_{i,i} \neq 0$, hence $b_{i,j} = 0$. Thus by induction, $B$ is upper triangular.
Now to the main result. $UT_n(F)$ is nonempty since the identity matrix is upper triangular. Now let $A, B \in UT_n(F)$; then $B^{-1} \in UT_n(F)$ by Lemma 3, and $AB^{-1} \in UT_n(F)$ by Lemma 2. So by the subgroup criterion, $UT_n(F)$ is a subgroup of $GL_n(F)$.
