Search papers, labs, and topics across Lattice.
This study evaluates the efficacy of finite-difference (FD) methods for derivative computation in Physics-Informed Neural Networks (PINNs), contrasting them with traditional automatic differentiation (AD). The authors demonstrate that with appropriate step size calibration, FD not only matches AD in accuracy across various PDE benchmarks but also offers improved speed and reduced GPU memory usage. Additionally, they reveal that standard PyTorch autograd techniques are flawed for certain neural architectures, while their proposed stochastic FD variant significantly enhances performance on stationary problems.
Finite-difference methods can outperform automatic differentiation in PINNs, achieving faster computations with less memory while maintaining accuracy.
We systematically investigate finite-difference (FD) derivative computation in Physics-Informed Neural Networks (PINNs) as an alternative to automatic differentiation (AD). On three benchmark PDEs we show that, with a properly calibrated step size, FD matches AD in accuracy on every problem while running faster across the full tested batch-size range and using substantially less GPU memory, and that a stochastic variant we propose outperforms AD on a stationary problem. We further show that for neural architectures with inter-sample dependencies (e.g. BatchNorm, self-attention) the standard PyTorch autograd idiom is silently incorrect; the correct per-sample alternative is computationally infeasible at PINN-relevant batch sizes, while FD provides a forward-only approximation that is empirically an order of magnitude closer to the true per-sample derivative.
