The use of deep neural networks (DNNs) in various geophysical applications has increased dramatically in the last decade. For instance, recent applications of DNNs in electromagnetic (EM) methods include inversion of Controlled Source Electromagnetic (CSEM) \cite{Puzyrev2019}, or borehole resistivity measurements \cite{Shahriari2020}, to name a few. However, training a DNN requires a massive amount of data samples, which involves solving many forward problems. In addition, accurate forward solvers often demand fine grid computations \cite{Grayver2014,Key2011}, which may lead to a prohibitive computational cost. Thus, we must design rapid forward solvers of parametric partial differential equations (PDEs) to produce data sets required to efficiently train DNNs. This presentation proposes a forward solver of parametric PDEs. Based on the simple-to-implement $hp$-adaptive strategy proposed by Caro et al. \cite{Caro2022}, we extend it to parametric PDEs by using a multi-adaptive goal-oriented (MAGO) algorithm. By doing so, we build a single $hp$-adaptive mesh capable of resolving parametric forward problems. To do so, we define an error indicator that combines information from several finite element solutions simultaneously. In addition, to limit the computational costs, we precompute the stiffness matrices of the considered parametric PDE. The resulting $hp$-adapted mesh can accurately solve many finite element problems needed for training a DNN at a reduced computational cost.