- Motivated by recently reported magnetic-field induced topological phases in ultracold atoms and correlated Moiré materials, we investigate topological phase transitions in a minimal model consisting of interacting spinless fermions described by the Hofstadter model on a square lattice. For interacting lattice Hamiltonians in the presence of a commensurate magnetic flux it has been demonstrated that the quantized Hall conductivity is constrained by a Lieb-Schultz-Mattis (LSM)-type theorem due to magnetic translation symmetry. In this work, we revisit the validity of the theorem for such models and establish that a topological phase transition from a topological to a trivial insulating phase can be realized but must be accompanied by spontaneous magnetic translation symmetry breaking caused by charge ordering of the spinless fermions. To support our findings, the topological phase diagram for varying interaction strength is mapped out numerically with exact diagonalization for different flux quantum ratios and band fillings using symmetry indicators. We discuss our results in the context of the LSM-type theorem.