Differential Equations
Mathwrist White Paper Series

Copyright ©Mathwrist LLC 2023
(October 5, 2023)
Abstract

This document presents a technical overview of solving differential equations in Mathwrist’s C++ Numerical Programming Library (NPL). The problem class we are targetting includes initial value problems for ordinary differential equations (ODE) and initial value/boundary problems for 1-d parabolic linear partial differential equations (PDE).

1 Ordinary Differential Equation

1.1 Introduction

For a smooth function y(t) governed by an ordinary differential equation (ODE)

y(t)=f(t,y(t)),t[0,T]

, where f(t,y(t)) is continuous and satisfies Lipschitz condition wrt y, then y(t) has a unique solution y(t) for 0tT given initial value y(0)=y0. See [1] for the details of Lipschitz condition, proofs and various numerical methods.

To numerically solve an ODE initial value problem, we generate the approximation ui of y(ti) over a sequence of time steps ti,i=0,,N with u0=y0. Explicit one step methods have the general form,

ui+1=ui+hϕ(,ui),h=ti+1-ti

, where the evaluation of ϕ(,ui) depends only on the approximation value ui at the previous step ti. The simplest one-step method is Euler’s method, which has global relative error 𝒪(h) with step size h.

The family of Runge-Kutta methods are high order accurate 1-step methods, which require multiple evaluations of f(ti+Δt,ui+k) for some Δt[0,h] and some bump amount k. Evaluation at a mid-way point ti+Δt could be infeasible in practice. For example, f(t,y(t)) is an implicit function and can be computed only on discretized grid points [ti,ti+1,]. If f(ti+Δt,) indeed can be easily computed, Runge-Kutta methods usually perform better than multi-step methods.

Multi-step methods can also produce high order accuracy. A m-step multi-step method has the general form

ui+1 = αm-1ui+αm-2ui-1++α0ui+1-m+
h[βmf(ti+1,ui+1)+βm-1f(ti,ui)++β0f(ti+1-m,ui+1-m)]

If coefficient βm=0, the method is explicit or otherwise implicit. Explicit and implicit approaches can be combined to have the flavor of prediction and correction, i.e. ui+1 is first computed from the explicit method and then substituted to the implicit formula. Prediction and correction can be repeated until some error control tolerance is satisfied. Alternatively, prediction and correction can be used as error control to construct variable step-size methods.

The family of Adams-Bashforth methods and Adams-Moulton methods are all multi-step methods. The advantage of multi-step methods is that f(t,y(t)) is computed exactly on grid points and is computed only once at each ti. However, multi-step methods often face stability issue when h is not fine enough. For smooth problems though, multi-step methods could be excellent choices.

In the same spirit of using extrapolation to increase the accuracy of numerical integration, extrapolation can be used on top of the Euler’s method to increase the accuracy. This is a very powerful technique that is easy to implement and has the advantage of error control. See [1], section 5.8 for details.

In Mathwrist NPL, we choose an appropriate explicit or implicit ODE method as the base method and then increase the accuracy order of the base method within an Iterative Deferred Correction (IDeC) framework. Taking Euler’s method as the base method for example, our IDeC ODE solvers overall require less number evaluation of y(t) than that of the extrapolation method to achieve the same level of accuracy.

1.2 Iterative Deferred Correction (IDeC)

The idea of deferred correction on a low order ODE method to obtain high order accuracy can be traced back to [2] and [3]. Convergence proof and error order analysis are discussed in [4]. We shall call the low order ODE method the “base method”. IDeC gives one extra order accuracy on top of the base method almost for free. Applying multiple error correction iterations could then raise the accuracy order to a desired level.

1.2.1 Classic IDeC Method

Consider the i-th step of Euler’s method ui(1)=ui-1(1)+hf(ti-1,ui-1(1)) to solve the initial value problem

y(t)=f(t,y(t)),0tT,y(0)=y0 (1)

Here we use the notation u(1) to indicate that it is a first order approximation of y(t). Now construct a smooth interpolation function L(t) passing through p+1 number of points {ui-p(1),,ui(1)} that have been computed by Euler’s method. The error function e(t)=y(t)-L(t) then satisfies the ODE

e(t)=f(t,y(t))-L(t) (2)

Writing the ODE in terms of e(t) and L(t) using the relation y(t)=L(t)+e(t), the ODE is e(t)=g(t,e(t)), where g(t,e)=f(t,L(t)+e)-L(t). Given initial value e(0)=0, we can again solve e(t) using Euler’s method

ei+1=ei+hg(ti,ei)=ei+h[f(ti,ui(1)+ei)-L(ti)] (3)

Since Euler’s method produces order 𝒪(h) relative error, Euler approximation on e(t) then has error order 𝒪(h2). So adding correction ei to ui(1) produces a second order approximation,

ui(2)=ui(1)+ei

This process can be repeated to raise the accuracy order. In Mathwrist NPL, we use Lagrange interpolation polynomial to construct L(t). The degree p of the interpolation polynomial is restricted to be 3<p<7 for practical reasons.

Let q be the number of error correction iterations. We restrict q<p. Extra error correction iterations generally do not increase the accuracy but only adding numerical noise. Our explicit IDeC solvers also allow user to choose the base method to be either the explicit Euler’s method or the Adams-Bashforth 3-step method.

1.2.2 Spectral IDeC Method

We can rewrite ODE initial value problem (1) in an integral equation form,

y(t)=y0+0tf(s,y(s))𝑑s (4)

Assume we have obtained an order-n approximation u(n)(t) of y(t). An accuracy measure of this approximation can be defined by the residual function,

ϵ(t)=y0+0tf(s,u(n)(s))𝑑s-u(n)(t) (5)

Substituting the true error function e(t)=y(t)-u(n)(t) back to equation (4), we have

u(n)(t)+e(t)=y0+0tf(s,u(n)(s)+e(s))𝑑s

Then using the definition (5) to obtain a Picard integral equation

e(t)=0t[f(s,u(n)(s)+e(s))-f(s,u(n)(s))]𝑑s+ϵ(t) (6)

Let g(t,e)=f(t,u(n)(t)+e(t))-f(t,u(n)(t)) and simplify equation (6) to

e(t)=0tg(s,e(s))𝑑s+ϵ(t) (7)

Now for knot points ti[0,h],i=0,,m,t0=0 and tm=h, we derive the explicit Euler scheme for the Picard integral equation (7)

ei+1=ei+g(ti,ei)Δti+ϵi+1-ϵi (8)

The spectral IDeC method uses Gaussian quadrature to compute ϵi from equation (5). In Mathwrist NPL, we use a Legendre polynomial L(x(t)) of degree p to compute the quadrature, where function x(t) mapps t[0,h] to x[-1,1]. This choice has 2p-2 order of accuracy when computing ϵ(t).

In the spectral IDeC case, we restrict 3<p<12 and q<p. Same as classic IDeC, the base method could be either the explicit Euler’s method or the Adams-Bashforth 3-step method.

1.2.3 Implicit IDeC

Explicit ODE methods need to use small enough step size h in order for the numerical solution to be stable. Implicit methods could be used in IDeC to address the stability issue. In fact, for a well-posed linear ODE problem, implicit methods could be unconditionally stable.

A n-dimensional system of ODE is linear if it is in a general form,

y(t)=𝐀(t)y(t)+s(t) (9)

, where 𝐀(t) is a n×n matrix that only depends on t. s(t) is a n-dimensional external source function. When s(t)=0, the system is homogeneous. System (9) is stable when all eigen values of 𝐀(t) are negative. We shall assume the problems to be solved are well posed in stability.

Let 𝐲 and 𝐬 denote vector values of y(t) and s(t). Using implicit Euler as the base method of a classic IDeC, we have

𝐲i+1=𝐲i+h𝐲(ti+1)=𝐲i+h[𝐀i+1𝐲i+1+𝐬i+1]

, which is to solve the linear system

[𝐈-h𝐀i+1]𝐲i+1=𝐲i+h𝐬i+1

It can be shown that if all eigen values of 𝐀i+1 are negative, the implicit Euler method is unconditionally stable. In Mathwrist NPL, we offer the combination of classic IDeC or spectral IDeC with implicit Euler as the base method. This is particularly useful to solve stiff ODE problems.

2 One Dimensional Parabolic PDE

2.1 Introduction

We consider a two dimensional smooth function v(t,x) in a rectangle area x(a,b),t(0,T) that is governed by a general form of 1-d linear parabolic PDE

vt=f(t,x)2vx2+g(t,x)vx+h(t,x)v+s(t,x) (10)

. Functions f(t,x), g(t,x) and h(t,x) at the right hand side are coefficient functions. For well-posed problems, f(t,x)>0. Function s(t,x) at the end of the equation is the external source function. A PDE is called homogeneous when s(t,x)=0, or inhomogeneous otherwise.

We often refer t as the temporal variable and x as the spatial or state variable. 1-d parabolic PDE refers to the fact there is only one state variable x. In a linear parabolic PDE, all coefficient functions and the source function only depend on t and x. We will also use partial derivative notations vt,vx,vxx interchangeably with notations in (10) for convenience.

We could introduce a partial derivative operator 𝒜(v) wrt spatial variable and write the PDE as an ODE form

vt = 𝒜(v)+s(t,x) (11)
𝒜(v) = f(t,x)vxx+g(t,x)vx+h(t,x)v (12)

For the numerical methods that we will discuss in the following sections, operator 𝒜(v) translates to a matrix-vector multiplication 𝐀𝐯. For example, 𝐀 is constructed as the finite difference coefficient matrix by a finite difference scheme and 𝐯 is the vector of v(t,x) approximated at discretized node points.

For parabolic PDEs, the appropriate boundary conditions [8] are listed in table 1. Effectively, the Robin condition is a mixture of Dirichlet and Neumann conditions. Other boundary conditions lead to no solution.

Given an initial value condition v(0,x)=u(x) and appropriate boundary conditions, a parabolic PDE has a unique and stable solution. The solution of v(t,x) is immediately smoothed out for any t>0 even though the initial value v(0,x) could be non-smooth.

Table 1: Appropriate Boundary Conditions for Parabolic PDE
Dirichlet v(t,a)=u1(t)v(t,b)=u2(t)
Neumann vx(t,a)=u1(t)vx(t,b)=u2(t)
Robin α1v(t,a)+β1vx(t,a)=u1(t)α2v(t,b)+β2vx(t,b)=u2(t)

2.2 Crank-Nicolson Method

Rearrange the PDE form (10) to

vt=w(t,x,v,vx,vxx)+s(t,x) (13)

, where we express the 0-th to 2nd order spatial derivative terms in a new function w(t,x,v,vx,vxx). Consider a mesh {tl=0,,N}×{xj=0,,J} that is uniformly discretized in the spatial dimension, e.g. xj=a+ib-aJ. Denote the function values at grid point (tl,xj) as fl,j:=f(tl,xj), gl,j:=g(tl,xj), hl,j:=h(tl,xj), sl,j:=s(tl,xj), vl,j:=v(tl,xj) and wl,j:=w(tl,xj).

The Crank-Nicolson (CN) finite difference method approximates the spacial derivatives by center difference,

vx|t=tl,x=xj = vl,j+1-vl,j-12Δx
vxx|t=tl,x=xj = vl,j+1-2vl,j+vl,j-1Δx2

Along the time dimension, consider a middle point (t,xj),t(tl,tl+1). The CN method approximates time derivative by center difference and averages the right hand side of equation (13) as

vt|x=xj=vl+1,j-vl,jΔt=ϕ(wl+1,j+sl+1,j)+ψ(wl,j+sl,j),ψ=1-ϕ

CN method effectively sets ϕ=12. The above difference equation is a general form that has the flavor of choosing 0ϕ1 from fully explicit to fully implicit. Then differential equation (13) is approximated by the following finite difference equation,

vl+1,jΔt-
ϕ[fl+1,jvl+1,j+1-2vl+1,j+vl+1,j-1Δx2+gl+1,jvl+1,j+1-vl+1,j-12Δx+hl+1,jvl+1,j]=
vl,jΔt+ψ[fl,jvl,j+1-2vl,j+vl,j-1Δx2+gl,jvl,j+1-vl,j-12Δx+hl,jvl,j]+
ϕsl+1,j+ψsl,j

At boundary states e.g., x0, this difference equation involves node values vl,-1 and vl+1,-1. We can introduce “ghost point” x-1 and incorporate with boundary conditions. In the general form of Robin conditions, we can express vl,-1 as

vl,-1=2α1Δxβ1vl,0+vl,1-2Δxβ1u1(tl)

, thereby eliminate nodal values vl,-1 and vl+1,-1. For all states xj to satisfy the difference equations, effectively we solve a linear system of the follow type at each tl time step.

𝐁l+1vl+1=𝐀lvl+𝐬 (14)

The CN method has second order accuracy 𝒪(Δt2) and 𝒪(Δx2) in both time and space dimension. It is unconditionaly stable. Finite difference methods work well in the situation when initial value v(0,x) is not smooth or even non-differentiable. In which case, non-smooth v(0,x) will cause grid values oscillating at the first few time steps but soonly being smoothed out as t departs from 0. To improve accuracy, Mathwrist NPL allows users to specify an initial smoothing time s>0. Internally, we will start with a 4-th order accurate finite difference method using the method of lines to approximate v(t,x) to time t=s and then switch back to the regular CN method.

2.3 Spectral Methods

2.3.1 Introduction

Spectral methods approximate the solution v(t,x) by an expansion function u(t,x)=k=-ak(t)ϕk(x) in terms of trial basis functions ϕk(x). The basis functions are global orthogonal polynomials. If u(x) is a periodic function, the trial functions are typically choosen to be complex exponentials ϕk(x)=eikx. If u(x) is not periodic, ϕk(x) are usually choosen to be Chebyshev or Legendre polynomials. With these choices of basis functions, it can be shown that the approximation converges to the true solution at exponential rate, i.e. the coefficient series ak converges to 0 exponentially. This property allows us to use a truncated series approximation.

For simplicity, consider a homogeneous PDE vt=𝒜(v) and approximate v(t,x) by truncated series expansion,

un(t,x)=k=-n/2n/2ak(t)ϕk(x)

, where the notation un(t,x) indicates it is an approximation polynomial of degree n. Different spectral methods apply different techniques to reduce the residual term tun-𝒜(un). One approach, known as weak formulation, is to select a set of test functions ψk(x),k=-n2,,n2 and require

ab[tun+𝒜(un)]ψk(x)𝑑x=0,k

An alternative approach takes a set of collocation points xj(a,b) and requires the PDE being exactly satisfied at the collocation points,

tun|x=xj=𝒜(un),j

This approach is called spectral collocation method or pseudo-spectral method. It is a strong formulation because it perserves the original differential order. Spectral collocation methods are similar to finite difference methods in the sense that they all impose the strong formulation of the original problem to a set of discrete points.

2.3.2 Chebyshev Collocation

Mathwrist NPL implements a Chebyshev collocation method where we choose Chebyshev polynomials as basis functions

ϕk(x)=Tk(x)=cos(kcos-1x)

and collocation points being the extrema points of n-th order Chebyshev polynomial xj=cosπjn,j=0,,n. The approximation polynomial is

un(t,x)=k=0nak(t)Tk(x),k=0,,n (15)

Coefficients ak(t) can be expressed in terms of node values u(t,xj) [5]. We can then impose the strong form of the 1-d parabolic PDE in (10) to un(t,x) and obtain an ODE form,

unt|x=xj=f(t,x)2unx2+g(t,x)unx+h(t,x)un+s(t,x) (16)

Solving the PDE at collocation points then becomes to solve a system of ODE, where the spatial partial derivatives can be computed from (15) or its equivalent interpolation polynomial. At boundary points x0 and xn, nodal values u0 and un need incorporate with boundary conditions. When solving ODE system (16), Mathwrist NPL uses its spectral IDeC method with implicit Euler as the base method.

Not suprisingly, spectral collocation method outperforms when v(t,x) is very smooth in spatial dimension, including initial values v(0,x). When v(t,x) has fluctuations or strong curvature in states, users can specify the initial smoothing time parameter s like our CN method. In which case, Mathwrist NPL will internally use an higher order finite difference method of lines to approximate v(t,x) over the starting period 0<t<s and then switch to spectral collocation method by taking v(s,x) as initial values for the rest of time.

2.4 Method of Lines with Finite Difference

2.4.1 Introduction

The technique of method of lines (MOL) in general sense refers to the family of numerical algorithms that solve a parabolic PDE (10) by solving a system of ODE in the form of (11). Spectral collocation method of course falls into this category. Alternatively, we can approximate the spatial partial derivative operator 𝒜(v) by finite difference. Mathwrist NPL provides 2nd order and 4th order accurate finite difference scheme to approximate 𝒜(v). Classic IDeC methods are then used to solve the system of ODE with implicit Euler as the base method.

2.4.2 Fourth Order Finite Difference Method

Let δ(d)(v) be the d-th order finite difference operator wrt spatial state x at interior node xj. Using the method of undetermined coefficients, see [7] pp 19-20, at each grid point xk we approximate δ(1)(v)=vx and δ(2)(v)=2vx2 as linear combinations of grid values vj-2,vj-1,vj,vj+1,vj+2.

δ(d)(v)|x=xj = c0(d)vj-2+c1(d)vj-1+c2(d)vj+c3(d)vj+1+c4(d)vj+2

By matching the coefficients of derivative terms with the 4-th order Taylor expansion around xj, we can obtain

c0(1) = 112Δx,c1(1)=-23Δx,c2(1)=0,c3(1)=23Δx,c4(1)=-112Δx
c0(2) = -112Δx2,c1(2)=43Δx2,c2(2)=-52Δx2,c3(2)=43Δx2,c4(2)=-112Δx2

This is the 5-point center difference scheme, which approximates the first derivative vx and the second derivative vxx to 4th order accuracy for interior node xj,j=2,,J-2. For near boundary states x1 and xJ-1, alternative high order finite difference formulas are used. Lastly, high-order accurate schemes are used to approximate vx that appears in Neumann and Robin boundary conditions. Overall the method is 4th order accurate in states and relies on IDeC method to raise the accuracy along the time dimension, i.e. 3 error correction iterations to obtain 𝒪(h4) accuracy.

2.4.3 Second Order Finite Difference Method

We could choose the exactly same technique as the CN method in spatial discretization to approximate 𝒜(v) and boundary conditions. Along the time dimension, we then use implicit IDeC to solve the ODE system (11). Compared to the CN method, this approach offers users more freedom to choose spatial discretization Δx and time marching step size h. For example, the method affords to use a relatively larger step size h yet retain the same level of accuracy as the CN method.

References

  • [1] Richard L. Burden, J. Douglas Faires and Annette M. Burden: Numerical Analysis, 10th edition, Cengage Learning, 2016
  • [2] Pedro E. Zadunaisky: On the Estimation of Errors Propagated in the Numerical Integration of Ordinary Differential Equations, Numer. Math. 27, 21 - 39 (1976), Springer-Verlag 1976
  • [3] Alok Dutt, Leslie Greengard, Vladimir Rokhlin: Spectral Deferred Correction Methods for Ordinary Differential Equations, BIT, 40(2):241–266, 2000
  • [4] Anders C. Hansen, John Strain: On the Order of Deferred Correction, Applied Numerical Mathematics, Volume 61, Issue 8, August 2011, Pages 961-973
  • [5] C. Canuto, M.Y. Hussaini, A. Quarteroni and T. A. Zang: Spectral Methods, Fundamentals in Single Domains, Springer, 2006
  • [6] John P. Boyd: Chebyshev and Fourier Spectral Methods, Dover, 2nd edition, 2016
  • [7] J. W. Thomas: Numerical Partial Differential Equations: Finite Difference Methods, Springer, 2010
  • [8] Sandro Salsa: Partial Differential Equations in Action, from Modelling to Theory, Springer 2008