Tutorials, by Mathematical Problem#
TODO: Add link to Python example here
Linear elliptic PDE on a 2D grid#
WHAT THIS EXAMPLE DEMONSTRATES:
Using command line options
Using Linear Solvers
Handling a simple structured grid
FURTHER DETAILS:
DO THE FOLLOWING:
Compile
src/ksp/ksp/tutorials/ex50.c$ cd petsc/src/ksp/ksp/tutorials $ make ex50
Run a 1 processor example with a 3x3 mesh and view the matrix assembled
$ mpiexec -n 1 ./ex50 -da_grid_x 4 -da_grid_y 4 -mat_view
Expected output:
Mat Object: 1 MPI process type: seqaij row 0: (0, 0.) (1, 0.) (4, 0.) row 1: (0, 0.) (1, 0.) (2, 0.) (5, 0.) row 2: (1, 0.) (2, 0.) (3, 0.) (6, 0.) row 3: (2, 0.) (3, 0.) (7, 0.) row 4: (0, 0.) (4, 0.) (5, 0.) (8, 0.) row 5: (1, 0.) (4, 0.) (5, 0.) (6, 0.) (9, 0.) row 6: (2, 0.) (5, 0.) (6, 0.) (7, 0.) (10, 0.) row 7: (3, 0.) (6, 0.) (7, 0.) (11, 0.) row 8: (4, 0.) (8, 0.) (9, 0.) (12, 0.) row 9: (5, 0.) (8, 0.) (9, 0.) (10, 0.) (13, 0.) row 10: (6, 0.) (9, 0.) (10, 0.) (11, 0.) (14, 0.) row 11: (7, 0.) (10, 0.) (11, 0.) (15, 0.) row 12: (8, 0.) (12, 0.) (13, 0.) row 13: (9, 0.) (12, 0.) (13, 0.) (14, 0.) row 14: (10, 0.) (13, 0.) (14, 0.) (15, 0.) row 15: (11, 0.) (14, 0.) (15, 0.) Mat Object: 1 MPI process type: seqaij row 0: (0, 2.) (1, -1.) (4, -1.) row 1: (0, -1.) (1, 3.) (2, -1.) (5, -1.) row 2: (1, -1.) (2, 3.) (3, -1.) (6, -1.) row 3: (2, -1.) (3, 2.) (7, -1.) row 4: (0, -1.) (4, 3.) (5, -1.) (8, -1.) row 5: (1, -1.) (4, -1.) (5, 4.) (6, -1.) (9, -1.) row 6: (2, -1.) (5, -1.) (6, 4.) (7, -1.) (10, -1.) row 7: (3, -1.) (6, -1.) (7, 3.) (11, -1.) row 8: (4, -1.) (8, 3.) (9, -1.) (12, -1.) row 9: (5, -1.) (8, -1.) (9, 4.) (10, -1.) (13, -1.) row 10: (6, -1.) (9, -1.) (10, 4.) (11, -1.) (14, -1.) row 11: (7, -1.) (10, -1.) (11, 3.) (15, -1.) row 12: (8, -1.) (12, 2.) (13, -1.) row 13: (9, -1.) (12, -1.) (13, 3.) (14, -1.) row 14: (10, -1.) (13, -1.) (14, 3.) (15, -1.) row 15: (11, -1.) (14, -1.) (15, 2.)
Run with a 120x120 mesh on 4 processors using superlu_dist and view the solver options used
$ mpiexec -n 4 ./ex50 -da_grid_x 120 -da_grid_y 120 -pc_type lu -pc_factor_mat_solver_type superlu_dist -ksp_monitor -ksp_view
Expected output:
0 KSP Residual norm 3.039809126331e+00 1 KSP Residual norm 2.395703277441e-14 KSP Object: 4 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4 MPI processes type: lu out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: external factor fill ratio given 0., needed 0. Factored matrix follows: Mat Object: 4 MPI processes type: superlu_dist rows=14400, cols=14400 package used to perform factorization: superlu_dist total: nonzeros=0, allocated nonzeros=0 SuperLU_DIST run parameters: Process grid nprow 2 x npcol 2 Equilibrate matrix TRUE Replace tiny pivots FALSE Use iterative refinement FALSE Processors in row 2 col partition 2 Row permutation LargeDiag_MC64 Column permutation METIS_AT_PLUS_A Parallel symbolic factorization FALSE Repeated factorization SamePattern linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=14400, cols=14400 total: nonzeros=71520, allocated nonzeros=71520 total number of mallocs used during MatSetValues calls=0 has attached null space
Run with a 1025x1025 grid using multigrid solver on 4 processors with 9 multigrid levels
$ mpiexec -n 4 ./ex50 -da_grid_x 1025 -da_grid_y 1025 -pc_type mg -pc_mg_levels 9 -ksp_monitor
Expected output:
Nonlinear ODE arising from a time-dependent one-dimensional PDE#
WHAT THIS EXAMPLE DEMONSTRATES:
Using command line options
Handling a simple structured grid
Using the ODE integrator
Using call-back functions
FURTHER DETAILS:
DO THE FOLLOWING:
Compile
src/ts/tutorials/ex2.c$ cd petsc/src/ts/tutorials $ make ex2
Run a 1 processor example on the default grid with all the default solver options
$ mpiexec -n 1 ./ex2 -ts_max_steps 10 -ts_monitor
Expected output:
0 TS dt 0.00847458 time 0. 1 TS dt 0.00847458 time 0.00847458 2 TS dt 0.00847458 time 0.0169492 3 TS dt 0.00847458 time 0.0254237 4 TS dt 0.00847458 time 0.0338983 5 TS dt 0.00847458 time 0.0423729 6 TS dt 0.00847458 time 0.0508475 7 TS dt 0.00847458 time 0.059322 8 TS dt 0.00847458 time 0.0677966 9 TS dt 0.00847458 time 0.0762712 10 TS dt 0.00847458 time 0.0847458
Run with the same options on 4 processors plus monitor convergence of the nonlinear and linear solvers
$ mpiexec -n 4 ./ex2 -ts_max_steps 10 -ts_monitor -snes_monitor -ksp_monitor
Expected output:
0 TS dt 0.00847458 time 0. 0 SNES Function norm 1.044373877325e+01 0 KSP Residual norm 6.321628301999e-02 1 KSP Residual norm 6.526906698930e-03 2 KSP Residual norm 3.007791706552e-03 3 KSP Residual norm 4.405713554522e-04 4 KSP Residual norm 1.328208424515e-04 5 KSP Residual norm 1.644811353135e-05 6 KSP Residual norm 2.166566242194e-06 7 KSP Residual norm 1.586470130595e-16 1 SNES Function norm 7.337449917266e-04 0 KSP Residual norm 3.134728198020e-06 1 KSP Residual norm 3.906901802932e-07 2 KSP Residual norm 9.017310449502e-08 3 KSP Residual norm 3.016048824996e-08 4 KSP Residual norm 5.545058115802e-09 5 KSP Residual norm 1.173289407756e-09 6 KSP Residual norm 2.688949492463e-10 7 KSP Residual norm 6.425931432155e-21 2 SNES Function norm 5.661678516636e-11 1 TS dt 0.00847458 time 0.00847458 0 SNES Function norm 1.044514631097e+01 0 KSP Residual norm 6.331279355759e-02 1 KSP Residual norm 6.535292747665e-03 2 KSP Residual norm 2.987562112557e-03 3 KSP Residual norm 4.407960732509e-04 4 KSP Residual norm 1.310867990310e-04 5 KSP Residual norm 1.643512035814e-05 6 KSP Residual norm 2.157619515591e-06 7 KSP Residual norm 9.439033917013e-17 1 SNES Function norm 7.215537347936e-04 0 KSP Residual norm 3.094217119938e-06 1 KSP Residual norm 3.853396165643e-07 2 KSP Residual norm 8.869933627178e-08 3 KSP Residual norm 2.968861004222e-08 4 KSP Residual norm 5.442261222566e-09 5 KSP Residual norm 1.152789140987e-09 6 KSP Residual norm 2.628606842614e-10 7 KSP Residual norm 3.895271394296e-21 2 SNES Function norm 5.533643492917e-11 2 TS dt 0.00847458 time 0.0169492 0 SNES Function norm 1.044653068944e+01 0 KSP Residual norm 6.340852763106e-02 1 KSP Residual norm 6.543595844294e-03 2 KSP Residual norm 2.967488018227e-03 3 KSP Residual norm 4.410154015305e-04 4 KSP Residual norm 1.293935090965e-04 5 KSP Residual norm 1.642264605661e-05 6 KSP Residual norm 2.149306177473e-06 7 KSP Residual norm 1.303479711243e-16 1 SNES Function norm 7.096638555936e-04 0 KSP Residual norm 3.054510440042e-06 1 KSP Residual norm 3.801000498387e-07 2 KSP Residual norm 8.725839831972e-08 3 KSP Residual norm 2.922750507411e-08 4 KSP Residual norm 5.341779435787e-09 5 KSP Residual norm 1.132841361166e-09 6 KSP Residual norm 2.569880731718e-10 7 KSP Residual norm 5.983404585705e-21 2 SNES Function norm 5.236235205743e-11 3 TS dt 0.00847458 time 0.0254237 0 SNES Function norm 1.044789247558e+01 0 KSP Residual norm 6.350349403220e-02 1 KSP Residual norm 6.551817537718e-03 2 KSP Residual norm 2.947571742346e-03 3 KSP Residual norm 4.412293238928e-04 4 KSP Residual norm 1.277401751314e-04 5 KSP Residual norm 1.641065933559e-05 6 KSP Residual norm 2.141569821046e-06 7 KSP Residual norm 7.468211737013e-17 1 SNES Function norm 6.980655055866e-04 0 KSP Residual norm 3.015586472511e-06 1 KSP Residual norm 3.749683834693e-07 2 KSP Residual norm 8.584934528817e-08 3 KSP Residual norm 2.877684460640e-08 4 KSP Residual norm 5.243554447127e-09 5 KSP Residual norm 1.113425160917e-09 6 KSP Residual norm 2.512722872950e-10 7 KSP Residual norm 3.506711593308e-21 2 SNES Function norm 5.113451407763e-11 4 TS dt 0.00847458 time 0.0338983 0 SNES Function norm 1.044923221792e+01 0 KSP Residual norm 6.359770143981e-02 1 KSP Residual norm 6.559959330184e-03 2 KSP Residual norm 2.927815370907e-03 3 KSP Residual norm 4.414378305489e-04 4 KSP Residual norm 1.261259895261e-04 5 KSP Residual norm 1.639913030560e-05 6 KSP Residual norm 2.134357967380e-06 7 KSP Residual norm 6.760613069100e-17 1 SNES Function norm 6.867492280866e-04 0 KSP Residual norm 2.977424249989e-06 1 KSP Residual norm 3.699416246968e-07 2 KSP Residual norm 8.447126428579e-08 3 KSP Residual norm 2.833631246001e-08 4 KSP Residual norm 5.147530491106e-09 5 KSP Residual norm 1.094520636202e-09 6 KSP Residual norm 2.457086187479e-10 7 KSP Residual norm 7.230574224864e-21 2 SNES Function norm 4.752621462354e-11 5 TS dt 0.00847458 time 0.0423729 0 SNES Function norm 1.045055044742e+01 0 KSP Residual norm 6.369115842069e-02 1 KSP Residual norm 6.568022679059e-03 2 KSP Residual norm 2.908220767658e-03 3 KSP Residual norm 4.416409177985e-04 4 KSP Residual norm 1.245501371995e-04 5 KSP Residual norm 1.638803042750e-05 6 KSP Residual norm 2.127621812336e-06 7 KSP Residual norm 7.898425027421e-17 1 SNES Function norm 6.757059470783e-04 0 KSP Residual norm 2.940003517446e-06 1 KSP Residual norm 3.650168880980e-07 2 KSP Residual norm 8.312328182674e-08 3 KSP Residual norm 2.790560484320e-08 4 KSP Residual norm 5.053653328311e-09 5 KSP Residual norm 1.076108839046e-09 6 KSP Residual norm 2.402925346880e-10 7 KSP Residual norm 3.741394720880e-21 2 SNES Function norm 4.666130700591e-11 6 TS dt 0.00847458 time 0.0508475 0 SNES Function norm 1.045184767808e+01 0 KSP Residual norm 6.378387343082e-02 1 KSP Residual norm 6.576008998535e-03 2 KSP Residual norm 2.888789584447e-03 3 KSP Residual norm 4.418385875472e-04 4 KSP Residual norm 1.230117980237e-04 5 KSP Residual norm 1.637733246166e-05 6 KSP Residual norm 2.121315990116e-06 7 KSP Residual norm 9.102989092310e-17 1 SNES Function norm 6.649269539423e-04 0 KSP Residual norm 2.903304720284e-06 1 KSP Residual norm 3.601913872254e-07 2 KSP Residual norm 8.180453791505e-08 3 KSP Residual norm 2.748442911816e-08 4 KSP Residual norm 4.961868923662e-09 5 KSP Residual norm 1.058171714893e-09 6 KSP Residual norm 2.350195851927e-10 7 KSP Residual norm 4.964834968715e-21 2 SNES Function norm 4.471025129852e-11 7 TS dt 0.00847458 time 0.059322 0 SNES Function norm 1.045312440769e+01 0 KSP Residual norm 6.387585481648e-02 1 KSP Residual norm 6.583919661259e-03 2 KSP Residual norm 2.869523271170e-03 3 KSP Residual norm 4.420308468558e-04 4 KSP Residual norm 1.215101490543e-04 5 KSP Residual norm 1.636701041839e-05 6 KSP Residual norm 2.115398352263e-06 7 KSP Residual norm 6.417124125528e-17 1 SNES Function norm 6.544038785722e-04 0 KSP Residual norm 2.867308940893e-06 1 KSP Residual norm 3.554624318597e-07 2 KSP Residual norm 8.051422770910e-08 3 KSP Residual norm 2.707250399844e-08 4 KSP Residual norm 4.872125443546e-09 5 KSP Residual norm 1.040692056328e-09 6 KSP Residual norm 2.298855614716e-10 7 KSP Residual norm 3.140670460028e-21 2 SNES Function norm 4.058847439815e-11 8 TS dt 0.00847458 time 0.0677966 0 SNES Function norm 1.045438111843e+01 0 KSP Residual norm 6.396711081541e-02 1 KSP Residual norm 6.591755999885e-03 2 KSP Residual norm 2.850423085515e-03 3 KSP Residual norm 4.422177075225e-04 4 KSP Residual norm 1.200443665787e-04 5 KSP Residual norm 1.635703950965e-05 6 KSP Residual norm 2.109829761879e-06 7 KSP Residual norm 6.553582414264e-17 1 SNES Function norm 6.441286836585e-04 0 KSP Residual norm 2.831997900324e-06 1 KSP Residual norm 3.508274228910e-07 2 KSP Residual norm 7.925155398311e-08 3 KSP Residual norm 2.666955853517e-08 4 KSP Residual norm 4.784371891515e-09 5 KSP Residual norm 1.023653459714e-09 6 KSP Residual norm 2.248862935554e-10 7 KSP Residual norm 3.952784367220e-21 2 SNES Function norm 4.092050026541e-11 9 TS dt 0.00847458 time 0.0762712 0 SNES Function norm 1.045561827745e+01 0 KSP Residual norm 6.405764955798e-02 1 KSP Residual norm 6.599519308575e-03 2 KSP Residual norm 2.831490102359e-03 3 KSP Residual norm 4.423991856931e-04 4 KSP Residual norm 1.186136279928e-04 5 KSP Residual norm 1.634739610156e-05 6 KSP Residual norm 2.104573899619e-06 7 KSP Residual norm 5.913882911408e-17 1 SNES Function norm 6.340936409061e-04 0 KSP Residual norm 2.797353906304e-06 1 KSP Residual norm 3.462838484911e-07 2 KSP Residual norm 7.801574775723e-08 3 KSP Residual norm 2.627533147918e-08 4 KSP Residual norm 4.698557254015e-09 5 KSP Residual norm 1.007040261358e-09 6 KSP Residual norm 2.200177803007e-10 7 KSP Residual norm 3.762004175411e-21 2 SNES Function norm 4.009143190166e-11 10 TS dt 0.00847458 time 0.0847458
Run with the same options on 4 processors with 128 grid points
$ mpiexec -n 16 ./ex2 -ts_max_steps 10 -ts_monitor -M 128
Expected output:
0 TS dt 0.00393701 time 0. 1 TS dt 0.00393701 time 0.00393701 2 TS dt 0.00393701 time 0.00787402 3 TS dt 0.00393701 time 0.011811 4 TS dt 0.00393701 time 0.015748 5 TS dt 0.00393701 time 0.019685 6 TS dt 0.00393701 time 0.023622 7 TS dt 0.00393701 time 0.0275591 8 TS dt 0.00393701 time 0.0314961 9 TS dt 0.00393701 time 0.0354331 10 TS dt 0.00393701 time 0.0393701
Nonlinear PDE on a structured grid#
WHAT THIS EXAMPLE DEMONSTRATES:
Handling a 2d structured grid
Using the nonlinear solvers
Changing the default linear solver
FURTHER DETAILS:
DO THE FOLLOWING:
Compile
src/snes/tutorials/ex19.c$ cd petsc/src/snes/tutorials/ $ make ex19
Run a 4 processor example with 5 levels of grid refinement, monitor the convergence of the nonlinear and linear solver and examine the exact solver used
$ mpiexec -n 4 ./ex19 -da_refine 5 -snes_monitor -ksp_monitor -snes_view
Expected output:
lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. 0 SNES Function norm 1.036007954337e-02 0 KSP Residual norm 9.144944502871e-02 1 KSP Residual norm 2.593759906204e-02 2 KSP Residual norm 1.669815200495e-02 3 KSP Residual norm 1.510777951698e-02 4 KSP Residual norm 1.458401237884e-02 5 KSP Residual norm 1.418635322926e-02 6 KSP Residual norm 1.377436725003e-02 7 KSP Residual norm 1.332236907186e-02 8 KSP Residual norm 1.288602527920e-02 9 KSP Residual norm 1.240018288138e-02 10 KSP Residual norm 1.186798872492e-02 11 KSP Residual norm 1.126565820237e-02 12 KSP Residual norm 1.063916289485e-02 13 KSP Residual norm 9.975377414549e-03 14 KSP Residual norm 9.354874256053e-03 15 KSP Residual norm 8.779095086508e-03 16 KSP Residual norm 8.257220099779e-03 17 KSP Residual norm 7.721504294489e-03 18 KSP Residual norm 7.165931362294e-03 19 KSP Residual norm 6.614579158468e-03 20 KSP Residual norm 6.069852590203e-03 21 KSP Residual norm 5.532583715678e-03 22 KSP Residual norm 5.012542062575e-03 23 KSP Residual norm 4.469698743551e-03 24 KSP Residual norm 3.946112491958e-03 25 KSP Residual norm 3.431223373793e-03 26 KSP Residual norm 2.968213186086e-03 27 KSP Residual norm 2.622089571760e-03 28 KSP Residual norm 2.418543969985e-03 29 KSP Residual norm 2.310824403854e-03 30 KSP Residual norm 2.236309852146e-03 31 KSP Residual norm 2.197075836614e-03 32 KSP Residual norm 2.184744877441e-03 33 KSP Residual norm 2.184139801213e-03 34 KSP Residual norm 2.177357542486e-03 35 KSP Residual norm 2.165221418018e-03 36 KSP Residual norm 2.150741735309e-03 37 KSP Residual norm 2.135576712445e-03 38 KSP Residual norm 2.119308727352e-03 39 KSP Residual norm 2.106838711740e-03 40 KSP Residual norm 2.095968914022e-03 41 KSP Residual norm 2.087204866778e-03 42 KSP Residual norm 2.075535849832e-03 43 KSP Residual norm 2.056901852426e-03 44 KSP Residual norm 2.026534413197e-03 45 KSP Residual norm 1.986848633315e-03 46 KSP Residual norm 1.940721994942e-03 47 KSP Residual norm 1.883144499412e-03 48 KSP Residual norm 1.803002891048e-03 49 KSP Residual norm 1.710431551202e-03 50 KSP Residual norm 1.590925854814e-03 51 KSP Residual norm 1.451082323211e-03 52 KSP Residual norm 1.301330299614e-03 53 KSP Residual norm 1.182230084682e-03 54 KSP Residual norm 1.107680018677e-03 55 KSP Residual norm 1.066507531910e-03 56 KSP Residual norm 1.042227387049e-03 57 KSP Residual norm 1.018322560203e-03 58 KSP Residual norm 9.885570705392e-04 59 KSP Residual norm 9.548240372245e-04 60 KSP Residual norm 9.201938192905e-04 61 KSP Residual norm 9.067345082392e-04 62 KSP Residual norm 9.048664588409e-04 63 KSP Residual norm 9.048625048106e-04 64 KSP Residual norm 9.045786478382e-04 65 KSP Residual norm 9.025111529622e-04 66 KSP Residual norm 8.966677177120e-04 67 KSP Residual norm 8.874037824397e-04 68 KSP Residual norm 8.789188093430e-04 69 KSP Residual norm 8.717007141537e-04 70 KSP Residual norm 8.635394193756e-04 71 KSP Residual norm 8.549188412543e-04 72 KSP Residual norm 8.449007643802e-04 73 KSP Residual norm 8.306833639556e-04 74 KSP Residual norm 8.180866477839e-04 75 KSP Residual norm 8.062571853623e-04 76 KSP Residual norm 7.936883824218e-04 77 KSP Residual norm 7.780490917915e-04 78 KSP Residual norm 7.615878046973e-04 79 KSP Residual norm 7.442269316420e-04 80 KSP Residual norm 7.232115811673e-04 81 KSP Residual norm 6.988082322432e-04 82 KSP Residual norm 6.683230529966e-04 83 KSP Residual norm 6.362177125994e-04 84 KSP Residual norm 5.971822394607e-04 85 KSP Residual norm 5.563666831991e-04 86 KSP Residual norm 5.122378485297e-04 87 KSP Residual norm 4.641141819361e-04 88 KSP Residual norm 4.127674620013e-04 89 KSP Residual norm 3.767940694930e-04 90 KSP Residual norm 3.464891353455e-04 91 KSP Residual norm 3.328326373461e-04 92 KSP Residual norm 3.152057745476e-04 93 KSP Residual norm 3.046565182399e-04 94 KSP Residual norm 3.006541516591e-04 95 KSP Residual norm 2.987345416223e-04 96 KSP Residual norm 2.969106149950e-04 97 KSP Residual norm 2.959679010652e-04 98 KSP Residual norm 2.951910867025e-04 99 KSP Residual norm 2.941123180589e-04 100 KSP Residual norm 2.926765604512e-04 101 KSP Residual norm 2.900410156058e-04 102 KSP Residual norm 2.863626926022e-04 103 KSP Residual norm 2.810952824275e-04 104 KSP Residual norm 2.755878113867e-04 105 KSP Residual norm 2.712557488209e-04 106 KSP Residual norm 2.679296794391e-04 107 KSP Residual norm 2.642026368303e-04 108 KSP Residual norm 2.602137324935e-04 109 KSP Residual norm 2.541382700234e-04 110 KSP Residual norm 2.454344737954e-04 111 KSP Residual norm 2.335628910244e-04 112 KSP Residual norm 2.168004908802e-04 113 KSP Residual norm 1.993544334742e-04 114 KSP Residual norm 1.812661079898e-04 115 KSP Residual norm 1.614155068832e-04 116 KSP Residual norm 1.450176652465e-04 117 KSP Residual norm 1.271814682476e-04 118 KSP Residual norm 1.114893752683e-04 119 KSP Residual norm 1.016403410116e-04 120 KSP Residual norm 9.221659791770e-05 121 KSP Residual norm 8.791115165789e-05 122 KSP Residual norm 8.529594818471e-05 123 KSP Residual norm 8.439823024838e-05 124 KSP Residual norm 8.432330233590e-05 125 KSP Residual norm 8.432313969703e-05 126 KSP Residual norm 8.431433381217e-05 127 KSP Residual norm 8.424348778495e-05 128 KSP Residual norm 8.382806777182e-05 129 KSP Residual norm 8.337135217553e-05 130 KSP Residual norm 8.306671396769e-05 131 KSP Residual norm 8.299046396158e-05 132 KSP Residual norm 8.298022748488e-05 133 KSP Residual norm 8.296556620174e-05 134 KSP Residual norm 8.293318221137e-05 135 KSP Residual norm 8.289997195859e-05 136 KSP Residual norm 8.288650847461e-05 137 KSP Residual norm 8.287793944348e-05 138 KSP Residual norm 8.282009713924e-05 139 KSP Residual norm 8.231936644200e-05 140 KSP Residual norm 8.092917384457e-05 141 KSP Residual norm 7.810875275548e-05 142 KSP Residual norm 7.372335915736e-05 143 KSP Residual norm 6.920873807564e-05 144 KSP Residual norm 6.510777151187e-05 145 KSP Residual norm 6.142132751787e-05 146 KSP Residual norm 5.816161902635e-05 147 KSP Residual norm 5.516213050219e-05 148 KSP Residual norm 5.242284932630e-05 149 KSP Residual norm 4.986057648037e-05 150 KSP Residual norm 4.733288055568e-05 151 KSP Residual norm 4.601506226246e-05 152 KSP Residual norm 4.523121336508e-05 153 KSP Residual norm 4.507733287029e-05 154 KSP Residual norm 4.507097170108e-05 155 KSP Residual norm 4.506961467378e-05 156 KSP Residual norm 4.505696152433e-05 157 KSP Residual norm 4.501836545704e-05 158 KSP Residual norm 4.495626487199e-05 159 KSP Residual norm 4.490012973376e-05 160 KSP Residual norm 4.480136241466e-05 161 KSP Residual norm 4.458312996682e-05 162 KSP Residual norm 4.431767440291e-05 163 KSP Residual norm 4.407964983727e-05 164 KSP Residual norm 4.383992113924e-05 165 KSP Residual norm 4.357188317772e-05 166 KSP Residual norm 4.319522376980e-05 167 KSP Residual norm 4.261206561683e-05 168 KSP Residual norm 4.205899613192e-05 169 KSP Residual norm 4.138430977164e-05 170 KSP Residual norm 4.047346042359e-05 171 KSP Residual norm 3.933307539335e-05 172 KSP Residual norm 3.819767627834e-05 173 KSP Residual norm 3.702517997681e-05 174 KSP Residual norm 3.583191921804e-05 175 KSP Residual norm 3.458798761674e-05 176 KSP Residual norm 3.316083374306e-05 177 KSP Residual norm 3.173967896731e-05 178 KSP Residual norm 3.016354626802e-05 179 KSP Residual norm 2.866779750173e-05 180 KSP Residual norm 2.702938196877e-05 181 KSP Residual norm 2.618361138750e-05 182 KSP Residual norm 2.522495437254e-05 183 KSP Residual norm 2.426023897276e-05 184 KSP Residual norm 2.355948721907e-05 185 KSP Residual norm 2.319684487218e-05 186 KSP Residual norm 2.289784420766e-05 187 KSP Residual norm 2.267598687625e-05 188 KSP Residual norm 2.240641749204e-05 189 KSP Residual norm 2.212551730336e-05 190 KSP Residual norm 2.170264854588e-05 191 KSP Residual norm 2.112756030054e-05 192 KSP Residual norm 2.038822399814e-05 193 KSP Residual norm 1.962951220216e-05 194 KSP Residual norm 1.884493949304e-05 195 KSP Residual norm 1.799734026963e-05 196 KSP Residual norm 1.722254569823e-05 197 KSP Residual norm 1.660423842819e-05 198 KSP Residual norm 1.621056066730e-05 199 KSP Residual norm 1.591018158958e-05 200 KSP Residual norm 1.557926981647e-05 201 KSP Residual norm 1.510189268164e-05 202 KSP Residual norm 1.440759642876e-05 203 KSP Residual norm 1.349458967348e-05 204 KSP Residual norm 1.240308276374e-05 205 KSP Residual norm 1.118091740362e-05 206 KSP Residual norm 9.955874799398e-06 207 KSP Residual norm 8.667314210234e-06 208 KSP Residual norm 7.389939064823e-06 209 KSP Residual norm 6.261620050378e-06 210 KSP Residual norm 5.246555512523e-06 211 KSP Residual norm 4.721004890241e-06 212 KSP Residual norm 4.239837116741e-06 213 KSP Residual norm 3.816477467422e-06 214 KSP Residual norm 3.501683693279e-06 215 KSP Residual norm 3.305190215185e-06 216 KSP Residual norm 3.206138813817e-06 217 KSP Residual norm 3.174323738414e-06 218 KSP Residual norm 3.169528835126e-06 219 KSP Residual norm 3.169521851846e-06 220 KSP Residual norm 3.165241221321e-06 221 KSP Residual norm 3.145015122355e-06 222 KSP Residual norm 3.096044377523e-06 223 KSP Residual norm 3.018842023098e-06 224 KSP Residual norm 2.964634266861e-06 225 KSP Residual norm 2.957729966340e-06 226 KSP Residual norm 2.953877433705e-06 227 KSP Residual norm 2.925464755647e-06 228 KSP Residual norm 2.868821700731e-06 229 KSP Residual norm 2.782027517577e-06 230 KSP Residual norm 2.646127535134e-06 231 KSP Residual norm 2.482650898676e-06 232 KSP Residual norm 2.309998463210e-06 233 KSP Residual norm 2.154086486854e-06 234 KSP Residual norm 2.002548899717e-06 235 KSP Residual norm 1.885163787351e-06 236 KSP Residual norm 1.820671950047e-06 237 KSP Residual norm 1.781332450628e-06 238 KSP Residual norm 1.751510777513e-06 239 KSP Residual norm 1.723392579686e-06 240 KSP Residual norm 1.694083934428e-06 241 KSP Residual norm 1.677189950467e-06 242 KSP Residual norm 1.673111374168e-06 243 KSP Residual norm 1.671932556435e-06 244 KSP Residual norm 1.670372344826e-06 245 KSP Residual norm 1.668281814293e-06 246 KSP Residual norm 1.664401756910e-06 247 KSP Residual norm 1.655699903087e-06 248 KSP Residual norm 1.644879786465e-06 249 KSP Residual norm 1.638313410510e-06 250 KSP Residual norm 1.634433920669e-06 251 KSP Residual norm 1.632288417000e-06 252 KSP Residual norm 1.630803349524e-06 253 KSP Residual norm 1.629108012046e-06 254 KSP Residual norm 1.625738527055e-06 255 KSP Residual norm 1.620864125655e-06 256 KSP Residual norm 1.616268872661e-06 257 KSP Residual norm 1.611801733029e-06 258 KSP Residual norm 1.602497312803e-06 259 KSP Residual norm 1.575738098262e-06 260 KSP Residual norm 1.519104509227e-06 261 KSP Residual norm 1.431857168315e-06 262 KSP Residual norm 1.338744222187e-06 263 KSP Residual norm 1.262050256308e-06 264 KSP Residual norm 1.193091353420e-06 265 KSP Residual norm 1.139031579299e-06 266 KSP Residual norm 1.088471447383e-06 267 KSP Residual norm 1.035130328689e-06 268 KSP Residual norm 9.777098693277e-07 269 KSP Residual norm 9.155430511512e-07 270 KSP Residual norm 8.568922899017e-07 1 SNES Function norm 1.178751989614e-06 0 KSP Residual norm 1.748815715091e-06 1 KSP Residual norm 1.624620439395e-06 2 KSP Residual norm 1.456422465392e-06 3 KSP Residual norm 1.326030517472e-06 4 KSP Residual norm 1.134584001300e-06 5 KSP Residual norm 9.824370585989e-07 6 KSP Residual norm 8.882499873515e-07 7 KSP Residual norm 8.249609129314e-07 8 KSP Residual norm 7.838632525267e-07 9 KSP Residual norm 7.558367537184e-07 10 KSP Residual norm 7.352641428514e-07 11 KSP Residual norm 7.177890029352e-07 12 KSP Residual norm 7.027407189224e-07 13 KSP Residual norm 6.870092629142e-07 14 KSP Residual norm 6.712782681272e-07 15 KSP Residual norm 6.556064942447e-07 16 KSP Residual norm 6.413026840450e-07 17 KSP Residual norm 6.250492096162e-07 18 KSP Residual norm 6.087611627271e-07 19 KSP Residual norm 5.930996641661e-07 20 KSP Residual norm 5.781788025672e-07 21 KSP Residual norm 5.610549351106e-07 22 KSP Residual norm 5.401956055125e-07 23 KSP Residual norm 5.168528059550e-07 24 KSP Residual norm 4.913547400553e-07 25 KSP Residual norm 4.653072018102e-07 26 KSP Residual norm 4.457206633372e-07 27 KSP Residual norm 4.263897743643e-07 28 KSP Residual norm 4.072207343179e-07 29 KSP Residual norm 3.820129426326e-07 30 KSP Residual norm 3.524926057079e-07 31 KSP Residual norm 3.348441637200e-07 32 KSP Residual norm 3.208234358783e-07 33 KSP Residual norm 3.080653563509e-07 34 KSP Residual norm 2.969658623379e-07 35 KSP Residual norm 2.873838557550e-07 36 KSP Residual norm 2.784716738215e-07 37 KSP Residual norm 2.682605079585e-07 38 KSP Residual norm 2.582627271510e-07 39 KSP Residual norm 2.471851529652e-07 40 KSP Residual norm 2.364422933814e-07 41 KSP Residual norm 2.262205960981e-07 42 KSP Residual norm 2.177174676790e-07 43 KSP Residual norm 2.119314681575e-07 44 KSP Residual norm 2.086939643300e-07 45 KSP Residual norm 2.069589867774e-07 46 KSP Residual norm 2.051651483193e-07 47 KSP Residual norm 2.028925351652e-07 48 KSP Residual norm 1.994371389840e-07 49 KSP Residual norm 1.960711107273e-07 50 KSP Residual norm 1.934148114236e-07 51 KSP Residual norm 1.905853168179e-07 52 KSP Residual norm 1.867410083506e-07 53 KSP Residual norm 1.820712301605e-07 54 KSP Residual norm 1.769911631032e-07 55 KSP Residual norm 1.726229506451e-07 56 KSP Residual norm 1.684625473534e-07 57 KSP Residual norm 1.621405677874e-07 58 KSP Residual norm 1.531432035392e-07 59 KSP Residual norm 1.424389275969e-07 60 KSP Residual norm 1.310183104254e-07 61 KSP Residual norm 1.242656204932e-07 62 KSP Residual norm 1.179240873971e-07 63 KSP Residual norm 1.123101522446e-07 64 KSP Residual norm 1.054132720446e-07 65 KSP Residual norm 9.849626843795e-08 66 KSP Residual norm 9.197247865719e-08 67 KSP Residual norm 8.638142369566e-08 68 KSP Residual norm 8.305351990631e-08 69 KSP Residual norm 8.136422492034e-08 70 KSP Residual norm 8.002361575492e-08 71 KSP Residual norm 7.866116430480e-08 72 KSP Residual norm 7.722692892725e-08 73 KSP Residual norm 7.544381224969e-08 74 KSP Residual norm 7.354404089853e-08 75 KSP Residual norm 7.119435912763e-08 76 KSP Residual norm 6.867518671189e-08 77 KSP Residual norm 6.602411006203e-08 78 KSP Residual norm 6.337351313821e-08 79 KSP Residual norm 6.080224058330e-08 80 KSP Residual norm 5.890231340688e-08 81 KSP Residual norm 5.776735028390e-08 82 KSP Residual norm 5.681661116681e-08 83 KSP Residual norm 5.569418042042e-08 84 KSP Residual norm 5.415755708381e-08 85 KSP Residual norm 5.233260646242e-08 86 KSP Residual norm 5.021186255249e-08 87 KSP Residual norm 4.793874560013e-08 88 KSP Residual norm 4.567033260035e-08 89 KSP Residual norm 4.349813734988e-08 90 KSP Residual norm 4.106532397968e-08 91 KSP Residual norm 3.953311916974e-08 92 KSP Residual norm 3.782549069823e-08 93 KSP Residual norm 3.618014409136e-08 94 KSP Residual norm 3.473408466117e-08 95 KSP Residual norm 3.295298494632e-08 96 KSP Residual norm 3.116362368283e-08 97 KSP Residual norm 2.964729931235e-08 98 KSP Residual norm 2.843092504497e-08 99 KSP Residual norm 2.752698726769e-08 100 KSP Residual norm 2.677449961663e-08 101 KSP Residual norm 2.623140990700e-08 102 KSP Residual norm 2.574444690034e-08 103 KSP Residual norm 2.522405798326e-08 104 KSP Residual norm 2.457253380175e-08 105 KSP Residual norm 2.370264120442e-08 106 KSP Residual norm 2.262668411404e-08 107 KSP Residual norm 2.152259489644e-08 108 KSP Residual norm 2.036137945628e-08 109 KSP Residual norm 1.909454196392e-08 110 KSP Residual norm 1.787445336551e-08 111 KSP Residual norm 1.667394027353e-08 112 KSP Residual norm 1.559924659359e-08 113 KSP Residual norm 1.481283115471e-08 114 KSP Residual norm 1.435158522296e-08 115 KSP Residual norm 1.414878343957e-08 116 KSP Residual norm 1.407393287977e-08 117 KSP Residual norm 1.404447798640e-08 118 KSP Residual norm 1.403314447128e-08 119 KSP Residual norm 1.402765026958e-08 120 KSP Residual norm 1.402395814514e-08 121 KSP Residual norm 1.402351651527e-08 122 KSP Residual norm 1.402351613398e-08 123 KSP Residual norm 1.402327634838e-08 124 KSP Residual norm 1.402226170240e-08 125 KSP Residual norm 1.402020560080e-08 126 KSP Residual norm 1.401585203411e-08 127 KSP Residual norm 1.400213674922e-08 128 KSP Residual norm 1.396738912660e-08 129 KSP Residual norm 1.392184633617e-08 130 KSP Residual norm 1.386006552773e-08 131 KSP Residual norm 1.378213258746e-08 132 KSP Residual norm 1.368010395178e-08 133 KSP Residual norm 1.353666156547e-08 134 KSP Residual norm 1.329698812650e-08 135 KSP Residual norm 1.293880270293e-08 136 KSP Residual norm 1.237318583853e-08 137 KSP Residual norm 1.179161581684e-08 138 KSP Residual norm 1.134206743886e-08 139 KSP Residual norm 1.099888749064e-08 140 KSP Residual norm 1.066259659918e-08 141 KSP Residual norm 1.032068623282e-08 142 KSP Residual norm 9.954236804205e-09 143 KSP Residual norm 9.560912215756e-09 144 KSP Residual norm 9.145545126143e-09 145 KSP Residual norm 8.738420927086e-09 146 KSP Residual norm 8.409260212909e-09 147 KSP Residual norm 8.127321870356e-09 148 KSP Residual norm 7.898748854589e-09 149 KSP Residual norm 7.703865149553e-09 150 KSP Residual norm 7.486870703879e-09 151 KSP Residual norm 7.330560146692e-09 152 KSP Residual norm 7.128689079801e-09 153 KSP Residual norm 6.964286032888e-09 154 KSP Residual norm 6.785137395884e-09 155 KSP Residual norm 6.640351575110e-09 156 KSP Residual norm 6.511156528025e-09 157 KSP Residual norm 6.423541298314e-09 158 KSP Residual norm 6.345751133879e-09 159 KSP Residual norm 6.280901533539e-09 160 KSP Residual norm 6.207943324992e-09 161 KSP Residual norm 6.142867578714e-09 162 KSP Residual norm 6.087418250155e-09 163 KSP Residual norm 6.034648495317e-09 164 KSP Residual norm 5.977088198390e-09 165 KSP Residual norm 5.912054132468e-09 166 KSP Residual norm 5.836856906294e-09 167 KSP Residual norm 5.765552516298e-09 168 KSP Residual norm 5.679756378903e-09 169 KSP Residual norm 5.563707770725e-09 170 KSP Residual norm 5.405011361483e-09 171 KSP Residual norm 5.189986018558e-09 172 KSP Residual norm 4.908102634804e-09 173 KSP Residual norm 4.706174551357e-09 174 KSP Residual norm 4.546685413987e-09 175 KSP Residual norm 4.435455313583e-09 176 KSP Residual norm 4.332523827655e-09 177 KSP Residual norm 4.203498282754e-09 178 KSP Residual norm 4.058340805179e-09 179 KSP Residual norm 3.912110698046e-09 180 KSP Residual norm 3.754156543395e-09 181 KSP Residual norm 3.673186018443e-09 182 KSP Residual norm 3.613029839620e-09 183 KSP Residual norm 3.561927158378e-09 184 KSP Residual norm 3.500712553539e-09 185 KSP Residual norm 3.426672494749e-09 186 KSP Residual norm 3.351413827965e-09 187 KSP Residual norm 3.271649485418e-09 188 KSP Residual norm 3.188154724333e-09 189 KSP Residual norm 3.100224644403e-09 190 KSP Residual norm 3.010790734288e-09 191 KSP Residual norm 2.899860376120e-09 192 KSP Residual norm 2.795307867707e-09 193 KSP Residual norm 2.656859285349e-09 194 KSP Residual norm 2.478450141125e-09 195 KSP Residual norm 2.312417968425e-09 196 KSP Residual norm 2.135819348150e-09 197 KSP Residual norm 1.987345509540e-09 198 KSP Residual norm 1.882308219422e-09 199 KSP Residual norm 1.824720724119e-09 200 KSP Residual norm 1.793394737731e-09 201 KSP Residual norm 1.777603206282e-09 202 KSP Residual norm 1.767124708984e-09 203 KSP Residual norm 1.759460100944e-09 204 KSP Residual norm 1.754016726123e-09 205 KSP Residual norm 1.750274824009e-09 206 KSP Residual norm 1.747607840035e-09 207 KSP Residual norm 1.744480855293e-09 208 KSP Residual norm 1.741655356119e-09 209 KSP Residual norm 1.739290578624e-09 210 KSP Residual norm 1.734739843580e-09 211 KSP Residual norm 1.731591096986e-09 212 KSP Residual norm 1.728201695680e-09 213 KSP Residual norm 1.723790864899e-09 214 KSP Residual norm 1.713216135283e-09 215 KSP Residual norm 1.699328282646e-09 216 KSP Residual norm 1.678451518644e-09 217 KSP Residual norm 1.657901783704e-09 218 KSP Residual norm 1.635934083952e-09 219 KSP Residual norm 1.612202184752e-09 220 KSP Residual norm 1.567594145713e-09 221 KSP Residual norm 1.512642314015e-09 222 KSP Residual norm 1.457761603944e-09 223 KSP Residual norm 1.408111435897e-09 224 KSP Residual norm 1.356076746727e-09 225 KSP Residual norm 1.307962107684e-09 226 KSP Residual norm 1.266646107226e-09 227 KSP Residual norm 1.231345481628e-09 228 KSP Residual norm 1.187725733507e-09 229 KSP Residual norm 1.141044747451e-09 230 KSP Residual norm 1.099737269841e-09 231 KSP Residual norm 1.066786397851e-09 232 KSP Residual norm 1.030129878172e-09 233 KSP Residual norm 9.927434935483e-10 234 KSP Residual norm 9.473773325131e-10 235 KSP Residual norm 9.089690854957e-10 236 KSP Residual norm 8.759516453077e-10 237 KSP Residual norm 8.535012664712e-10 238 KSP Residual norm 8.308754136837e-10 239 KSP Residual norm 8.082501666452e-10 240 KSP Residual norm 7.754022320857e-10 241 KSP Residual norm 7.572112123056e-10 242 KSP Residual norm 7.442389885537e-10 243 KSP Residual norm 7.283799305392e-10 244 KSP Residual norm 7.073231969200e-10 245 KSP Residual norm 6.852558048466e-10 246 KSP Residual norm 6.637193841945e-10 247 KSP Residual norm 6.457535438239e-10 248 KSP Residual norm 6.348852218182e-10 249 KSP Residual norm 6.257254477629e-10 250 KSP Residual norm 6.158564534747e-10 251 KSP Residual norm 6.053446723415e-10 252 KSP Residual norm 5.965943232727e-10 253 KSP Residual norm 5.889879966631e-10 254 KSP Residual norm 5.798723091489e-10 255 KSP Residual norm 5.693837266918e-10 256 KSP Residual norm 5.572444317590e-10 257 KSP Residual norm 5.444694016783e-10 258 KSP Residual norm 5.294254215977e-10 259 KSP Residual norm 5.111579131674e-10 260 KSP Residual norm 4.907812169770e-10 261 KSP Residual norm 4.698943704786e-10 262 KSP Residual norm 4.516736830848e-10 263 KSP Residual norm 4.344099614072e-10 264 KSP Residual norm 4.205651998682e-10 265 KSP Residual norm 4.094400044417e-10 266 KSP Residual norm 4.007111452005e-10 267 KSP Residual norm 3.933386697206e-10 268 KSP Residual norm 3.859019001818e-10 269 KSP Residual norm 3.778293647154e-10 270 KSP Residual norm 3.685811905503e-10 271 KSP Residual norm 3.629152212256e-10 272 KSP Residual norm 3.564840338194e-10 273 KSP Residual norm 3.500495329589e-10 274 KSP Residual norm 3.431898068167e-10 275 KSP Residual norm 3.353565364446e-10 276 KSP Residual norm 3.263783927727e-10 277 KSP Residual norm 3.192763021450e-10 278 KSP Residual norm 3.122085426690e-10 279 KSP Residual norm 3.052760168605e-10 280 KSP Residual norm 2.980518819939e-10 281 KSP Residual norm 2.912439358242e-10 282 KSP Residual norm 2.849441584275e-10 283 KSP Residual norm 2.782808821251e-10 284 KSP Residual norm 2.710684441319e-10 285 KSP Residual norm 2.633874526828e-10 286 KSP Residual norm 2.553129453009e-10 287 KSP Residual norm 2.463101545110e-10 288 KSP Residual norm 2.377718932698e-10 289 KSP Residual norm 2.297278980842e-10 290 KSP Residual norm 2.216396694908e-10 291 KSP Residual norm 2.140852462454e-10 292 KSP Residual norm 2.058870942400e-10 293 KSP Residual norm 1.971167887405e-10 294 KSP Residual norm 1.884340894074e-10 295 KSP Residual norm 1.791613990270e-10 296 KSP Residual norm 1.688633513441e-10 297 KSP Residual norm 1.602704727320e-10 298 KSP Residual norm 1.541392323809e-10 299 KSP Residual norm 1.504896772477e-10 300 KSP Residual norm 1.483487862182e-10 301 KSP Residual norm 1.471763898232e-10 302 KSP Residual norm 1.458511285335e-10 303 KSP Residual norm 1.446834701991e-10 304 KSP Residual norm 1.422626457779e-10 305 KSP Residual norm 1.383238030475e-10 306 KSP Residual norm 1.340609360645e-10 307 KSP Residual norm 1.295566320698e-10 308 KSP Residual norm 1.254740222524e-10 309 KSP Residual norm 1.211504994547e-10 310 KSP Residual norm 1.174647456465e-10 311 KSP Residual norm 1.136881957093e-10 312 KSP Residual norm 1.097588329841e-10 313 KSP Residual norm 1.041565322506e-10 314 KSP Residual norm 9.702228215912e-11 315 KSP Residual norm 8.989484119100e-11 316 KSP Residual norm 8.190664018704e-11 317 KSP Residual norm 7.558818834442e-11 318 KSP Residual norm 6.957179676675e-11 319 KSP Residual norm 6.447734334845e-11 320 KSP Residual norm 6.116288878725e-11 321 KSP Residual norm 5.888979447543e-11 322 KSP Residual norm 5.764019417950e-11 323 KSP Residual norm 5.691872542062e-11 324 KSP Residual norm 5.660800677735e-11 325 KSP Residual norm 5.653667085624e-11 326 KSP Residual norm 5.653626356788e-11 327 KSP Residual norm 5.653235239410e-11 328 KSP Residual norm 5.653229638983e-11 329 KSP Residual norm 5.651625676369e-11 330 KSP Residual norm 5.646144096698e-11 331 KSP Residual norm 5.644627590998e-11 332 KSP Residual norm 5.644531324883e-11 333 KSP Residual norm 5.644497420702e-11 334 KSP Residual norm 5.644493580837e-11 335 KSP Residual norm 5.644470457969e-11 336 KSP Residual norm 5.644168745929e-11 337 KSP Residual norm 5.640053714959e-11 338 KSP Residual norm 5.626705339217e-11 339 KSP Residual norm 5.617632737039e-11 340 KSP Residual norm 5.610418525034e-11 341 KSP Residual norm 5.596338060631e-11 342 KSP Residual norm 5.565290829848e-11 343 KSP Residual norm 5.515768341466e-11 344 KSP Residual norm 5.442406254737e-11 345 KSP Residual norm 5.353934703450e-11 346 KSP Residual norm 5.271979463229e-11 347 KSP Residual norm 5.159719023218e-11 348 KSP Residual norm 5.050230595494e-11 349 KSP Residual norm 4.886604439750e-11 350 KSP Residual norm 4.701727019984e-11 351 KSP Residual norm 4.529669235590e-11 352 KSP Residual norm 4.321338221199e-11 353 KSP Residual norm 4.146974082227e-11 354 KSP Residual norm 3.989258303406e-11 355 KSP Residual norm 3.833287369436e-11 356 KSP Residual norm 3.696924118282e-11 357 KSP Residual norm 3.572876582097e-11 358 KSP Residual norm 3.466908142629e-11 359 KSP Residual norm 3.362405494859e-11 360 KSP Residual norm 3.248076664724e-11 361 KSP Residual norm 3.183961506680e-11 362 KSP Residual norm 3.105405824388e-11 363 KSP Residual norm 3.019347608164e-11 364 KSP Residual norm 2.915901620646e-11 365 KSP Residual norm 2.819671262899e-11 366 KSP Residual norm 2.737270822959e-11 367 KSP Residual norm 2.674106883731e-11 368 KSP Residual norm 2.627612843956e-11 369 KSP Residual norm 2.574335369953e-11 370 KSP Residual norm 2.523339840051e-11 371 KSP Residual norm 2.465910296336e-11 372 KSP Residual norm 2.407616725222e-11 373 KSP Residual norm 2.348589790024e-11 374 KSP Residual norm 2.305940230387e-11 375 KSP Residual norm 2.268699192687e-11 376 KSP Residual norm 2.227499543133e-11 377 KSP Residual norm 2.188320344640e-11 378 KSP Residual norm 2.152291481840e-11 379 KSP Residual norm 2.108875033860e-11 380 KSP Residual norm 2.062954609947e-11 381 KSP Residual norm 2.007204894177e-11 382 KSP Residual norm 1.941423359969e-11 383 KSP Residual norm 1.872835639533e-11 384 KSP Residual norm 1.794965148165e-11 385 KSP Residual norm 1.714082018695e-11 2 SNES Function norm 1.215516598864e-11 SNES Object: 4 MPI processes type: newtonls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=655 total number of function evaluations=3 norm schedule ALWAYS Jacobian is built using colored finite differences on a DMDA SNESLineSearch Object: 4 MPI processes type: bt interpolation: cubic alpha=1.000000e-04 maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=40 KSP Object: 4 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4 MPI processes type: bjacobi number of blocks = 4 Local solver information for first block is in the following KSP and PC objects on rank 0: Use -ksp_view ::ascii_info_detail to display information for all blocks KSP Object: (sub_) 1 MPI process type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (sub_) 1 MPI process type: ilu out-of-place factorization 0 levels of fill tolerance for zero pivot 2.22045e-14 matrix ordering: natural factor fill ratio given 1., needed 1. Factored matrix follows: Mat Object: (sub_) 1 MPI process type: seqaij rows=9604, cols=9604, bs=4 package used to perform factorization: petsc total: nonzeros=188944, allocated nonzeros=188944 using I-node routines: found 2401 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: (sub_) 1 MPI process type: seqaij rows=9604, cols=9604, bs=4 total: nonzeros=188944, allocated nonzeros=188944 total number of mallocs used during MatSetValues calls=0 using I-node routines: found 2401 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=37636, cols=37636, bs=4 total: nonzeros=746512, allocated nonzeros=746512 total number of mallocs used during MatSetValues calls=0 Number of SNES iterations = 2
Run with the same options but use geometric multigrid as the linear solver
$ mpiexec -n 4 ./ex19 -da_refine 5 -snes_monitor -ksp_monitor -snes_view -pc_type mg
Expected output:
lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. 0 SNES Function norm 1.036007954337e-02 0 KSP Residual norm 2.388589583549e+00 1 KSP Residual norm 5.715829806981e-01 2 KSP Residual norm 4.623679005936e-02 3 KSP Residual norm 1.143381177646e-02 4 KSP Residual norm 2.015139840224e-03 5 KSP Residual norm 4.356196119798e-04 6 KSP Residual norm 4.240953066710e-05 7 KSP Residual norm 8.848315297175e-06 1 SNES Function norm 9.854304971115e-06 0 KSP Residual norm 3.868049496775e-05 1 KSP Residual norm 7.693574326868e-06 2 KSP Residual norm 1.059429116239e-06 3 KSP Residual norm 4.004524784804e-07 4 KSP Residual norm 1.050186948327e-07 5 KSP Residual norm 5.073180513583e-08 6 KSP Residual norm 2.510513776297e-08 7 KSP Residual norm 1.211886495400e-08 8 KSP Residual norm 1.911963112131e-09 9 KSP Residual norm 3.005260864225e-10 2 SNES Function norm 3.117674497824e-10 0 KSP Residual norm 3.005042584730e-10 1 KSP Residual norm 1.120673922713e-10 2 KSP Residual norm 3.288439453292e-11 3 KSP Residual norm 5.822504321413e-12 4 KSP Residual norm 2.486684466178e-12 5 KSP Residual norm 1.198858055503e-12 6 KSP Residual norm 6.255669709502e-13 7 KSP Residual norm 1.544647758005e-13 8 KSP Residual norm 4.592122224907e-14 9 KSP Residual norm 4.984149547392e-15 10 KSP Residual norm 8.905129652955e-16 3 SNES Function norm 1.045594761851e-14 SNES Object: 4 MPI processes type: newtonls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=26 total number of function evaluations=4 norm schedule ALWAYS Jacobian is built using colored finite differences on a DMDA SNESLineSearch Object: 4 MPI processes type: bt interpolation: cubic alpha=1.000000e-04 maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=40 KSP Object: 4 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4 MPI processes type: mg type is MULTIPLICATIVE, levels=6 cycles=v Cycles per PCApply=1 Not using Galerkin computed coarse grid matrices Coarse grid solver -- level 0 ------------------------------- KSP Object: (mg_coarse_) 4 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_coarse_) 4 MPI processes type: redundant First (color=0) of 4 PCs follows KSP Object: (mg_coarse_redundant_) 1 MPI process type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_coarse_redundant_) 1 MPI process type: lu out-of-place factorization tolerance for zero pivot 2.22045e-14 using diagonal shift on blocks to prevent zero pivot [INBLOCKS] matrix ordering: nd factor fill ratio given 5., needed 1.875 Factored matrix follows: Mat Object: (mg_coarse_redundant_) 1 MPI process type: seqaij rows=64, cols=64, bs=4 package used to perform factorization: petsc total: nonzeros=1920, allocated nonzeros=1920 using I-node routines: found 16 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: 1 MPI process type: seqaij rows=64, cols=64, bs=4 total: nonzeros=1024, allocated nonzeros=1024 total number of mallocs used during MatSetValues calls=0 using I-node routines: found 16 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=64, cols=64, bs=4 total: nonzeros=1024, allocated nonzeros=1024 total number of mallocs used during MatSetValues calls=0 Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (mg_levels_1_) 4 MPI processes type: chebyshev Chebyshev polynomial of first kind eigenvalue targets used: min 0.148269, max 1.63095 eigenvalues estimated via gmres: min 0.144902, max 1.48269 eigenvalues estimated using gmres with transform: [0. 0.1; 0. 1.1] KSP Object: (mg_levels_1_esteig_) 4 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test estimating eigenvalues using noisy right hand side maximum iterations=2, nonzero initial guess tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_levels_1_) 4 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=196, cols=196, bs=4 total: nonzeros=3472, allocated nonzeros=3472 total number of mallocs used during MatSetValues calls=0 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (mg_levels_2_) 4 MPI processes type: chebyshev Chebyshev polynomial of first kind eigenvalue targets used: min 0.149178, max 1.64096 eigenvalues estimated via gmres: min 0.0843938, max 1.49178 eigenvalues estimated using gmres with transform: [0. 0.1; 0. 1.1] KSP Object: (mg_levels_2_esteig_) 4 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test estimating eigenvalues using noisy right hand side maximum iterations=2, nonzero initial guess tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_levels_2_) 4 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=676, cols=676, bs=4 total: nonzeros=12688, allocated nonzeros=12688 total number of mallocs used during MatSetValues calls=0 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (mg_levels_3_) 4 MPI processes type: chebyshev Chebyshev polynomial of first kind eigenvalue targets used: min 0.146454, max 1.61099 eigenvalues estimated via gmres: min 0.0659153, max 1.46454 eigenvalues estimated using gmres with transform: [0. 0.1; 0. 1.1] KSP Object: (mg_levels_3_esteig_) 4 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test estimating eigenvalues using noisy right hand side maximum iterations=2, nonzero initial guess tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_levels_3_) 4 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=2500, cols=2500, bs=4 total: nonzeros=48400, allocated nonzeros=48400 total number of mallocs used during MatSetValues calls=0 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 4 ------------------------------- KSP Object: (mg_levels_4_) 4 MPI processes type: chebyshev Chebyshev polynomial of first kind eigenvalue targets used: min 0.141089, max 1.55197 eigenvalues estimated via gmres: min 0.044097, max 1.41089 eigenvalues estimated using gmres with transform: [0. 0.1; 0. 1.1] KSP Object: (mg_levels_4_esteig_) 4 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test estimating eigenvalues using noisy right hand side maximum iterations=2, nonzero initial guess tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_levels_4_) 4 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=9604, cols=9604, bs=4 total: nonzeros=188944, allocated nonzeros=188944 total number of mallocs used during MatSetValues calls=0 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 5 ------------------------------- KSP Object: (mg_levels_5_) 4 MPI processes type: chebyshev Chebyshev polynomial of first kind eigenvalue targets used: min 0.127956, max 1.40751 eigenvalues estimated via gmres: min 0.0380398, max 1.27956 eigenvalues estimated using gmres with transform: [0. 0.1; 0. 1.1] KSP Object: (mg_levels_5_esteig_) 4 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10, initial guess is zero tolerances: relative=1e-12, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test estimating eigenvalues using noisy right hand side maximum iterations=2, nonzero initial guess tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_levels_5_) 4 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=37636, cols=37636, bs=4 total: nonzeros=746512, allocated nonzeros=746512 total number of mallocs used during MatSetValues calls=0 Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=37636, cols=37636, bs=4 total: nonzeros=746512, allocated nonzeros=746512 total number of mallocs used during MatSetValues calls=0 Number of SNES iterations = 3
Note this requires many fewer iterations than the default solver
Run with the same options but use algebraic multigrid (hypre’s BoomerAMG) as the linear solver
$ mpiexec -n 4 ./ex19 -da_refine 5 -snes_monitor -ksp_monitor -snes_view -pc_type hypre
Expected output:
lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. 0 SNES Function norm 1.036007954337e-02 1 SNES Function norm 3.201130549705e-06 2 SNES Function norm 2.738200342766e-10 3 SNES Function norm 1.086838199638e-14 Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 3 Number of SNES iterations = 3
Note this requires many fewer iterations than the default solver but requires more linear solver iterations than geometric multigrid.
Run with the same options but use the ML preconditioner from Trilinos
$ mpiexec -n 4 ./ex19 -da_refine 5 -snes_monitor -ksp_monitor -snes_view -pc_type ml
Expected output:
lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. 0 SNES Function norm 1.036007954337e-02 0 KSP Residual norm 4.888657920112e-01 1 KSP Residual norm 3.141627616286e-01 2 KSP Residual norm 2.358812378525e-01 3 KSP Residual norm 1.259045437573e-01 4 KSP Residual norm 7.408575133258e-02 5 KSP Residual norm 4.159658325629e-02 6 KSP Residual norm 2.181861584091e-02 7 KSP Residual norm 1.076244407256e-02 8 KSP Residual norm 5.242924144023e-03 9 KSP Residual norm 2.588726461570e-03 10 KSP Residual norm 1.400700093556e-03 11 KSP Residual norm 7.759158101290e-04 12 KSP Residual norm 4.655744671687e-04 13 KSP Residual norm 3.244240900196e-04 14 KSP Residual norm 2.327144448481e-04 15 KSP Residual norm 1.500951563875e-04 16 KSP Residual norm 8.327955132177e-05 17 KSP Residual norm 4.672468799190e-05 18 KSP Residual norm 2.889321996959e-05 19 KSP Residual norm 1.903514112532e-05 20 KSP Residual norm 1.298123631775e-05 21 KSP Residual norm 8.775705379722e-06 22 KSP Residual norm 5.330175482523e-06 23 KSP Residual norm 2.869775929854e-06 1 SNES Function norm 1.370527153606e-06 0 KSP Residual norm 1.177060567864e-05 1 KSP Residual norm 6.928128579778e-06 2 KSP Residual norm 4.304601930959e-06 3 KSP Residual norm 2.454875323346e-06 4 KSP Residual norm 1.512790232898e-06 5 KSP Residual norm 1.001915525185e-06 6 KSP Residual norm 7.387623107528e-07 7 KSP Residual norm 4.953711915641e-07 8 KSP Residual norm 3.144821641519e-07 9 KSP Residual norm 2.134819078071e-07 10 KSP Residual norm 1.518551249827e-07 11 KSP Residual norm 1.014185303569e-07 12 KSP Residual norm 5.608673008996e-08 13 KSP Residual norm 3.030519067648e-08 14 KSP Residual norm 1.786947998191e-08 15 KSP Residual norm 1.177816744649e-08 16 KSP Residual norm 8.997335866987e-09 17 KSP Residual norm 6.356106556680e-09 18 KSP Residual norm 4.147037399694e-09 19 KSP Residual norm 2.711154831563e-09 20 KSP Residual norm 1.750865415567e-09 21 KSP Residual norm 1.127681613823e-09 22 KSP Residual norm 7.783034009647e-10 23 KSP Residual norm 5.047971816405e-10 24 KSP Residual norm 3.137090395920e-10 25 KSP Residual norm 2.182629803057e-10 26 KSP Residual norm 1.265836293994e-10 27 KSP Residual norm 7.756916170515e-11 2 SNES Function norm 2.339734806142e-11 SNES Object: 4 MPI processes type: newtonls maximum iterations=50, maximum function evaluations=10000 tolerances: relative=1e-08, absolute=1e-50, solution=1e-08 total number of linear solver iterations=50 total number of function evaluations=3 norm schedule ALWAYS Jacobian is built using colored finite differences on a DMDA SNESLineSearch Object: 4 MPI processes type: bt interpolation: cubic alpha=1.000000e-04 maxstep=1.000000e+08, minlambda=1.000000e-12 tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08 maximum iterations=40 KSP Object: 4 MPI processes type: gmres restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement happy breakdown tolerance 1e-30 maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 4 MPI processes type: ml type is MULTIPLICATIVE, levels=5 cycles=v Cycles per PCApply=1 Using externally compute Galerkin coarse grid matrices Coarse grid solver -- level 0 ------------------------------- KSP Object: (mg_coarse_) 4 MPI processes type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_coarse_) 4 MPI processes type: redundant First (color=0) of 4 PCs follows KSP Object: (mg_coarse_redundant_) 1 MPI process type: preonly maximum iterations=10000, initial guess is zero tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_coarse_redundant_) 1 MPI process type: lu out-of-place factorization tolerance for zero pivot 2.22045e-14 using diagonal shift on blocks to prevent zero pivot [INBLOCKS] matrix ordering: nd factor fill ratio given 5., needed 1. Factored matrix follows: Mat Object: (mg_coarse_redundant_) 1 MPI process type: seqaij rows=16, cols=16, bs=4 package used to perform factorization: petsc total: nonzeros=256, allocated nonzeros=256 using I-node routines: found 4 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: 1 MPI process type: seqaij rows=16, cols=16, bs=4 total: nonzeros=256, allocated nonzeros=256 total number of mallocs used during MatSetValues calls=0 using I-node routines: found 4 nodes, limit used is 5 linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=16, cols=16, bs=4 total: nonzeros=256, allocated nonzeros=256 total number of mallocs used during MatSetValues calls=0 using I-node (on process 0) routines: found 1 nodes, limit used is 5 Down solver (pre-smoother) on level 1 ------------------------------- KSP Object: (mg_levels_1_) 4 MPI processes type: richardson damping factor=1. maximum iterations=2, nonzero initial guess tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_levels_1_) 4 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=88, cols=88, bs=4 total: nonzeros=4960, allocated nonzeros=4960 total number of mallocs used during MatSetValues calls=0 using I-node (on process 0) routines: found 4 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 2 ------------------------------- KSP Object: (mg_levels_2_) 4 MPI processes type: richardson damping factor=1. maximum iterations=2, nonzero initial guess tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_levels_2_) 4 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=764, cols=764, bs=4 total: nonzeros=35672, allocated nonzeros=35672 total number of mallocs used during MatSetValues calls=0 using I-node (on process 0) routines: found 50 nodes, limit used is 5 Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 3 ------------------------------- KSP Object: (mg_levels_3_) 4 MPI processes type: richardson damping factor=1. maximum iterations=2, nonzero initial guess tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_levels_3_) 4 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=7156, cols=7156, bs=4 total: nonzeros=259236, allocated nonzeros=259236 total number of mallocs used during MatSetValues calls=0 not using I-node (on process 0) routines Up solver (post-smoother) same as down solver (pre-smoother) Down solver (pre-smoother) on level 4 ------------------------------- KSP Object: (mg_levels_4_) 4 MPI processes type: richardson damping factor=1. maximum iterations=2, nonzero initial guess tolerances: relative=1e-05, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (mg_levels_4_) 4 MPI processes type: sor type = local_symmetric, iterations = 1, local iterations = 1, omega = 1. linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=37636, cols=37636, bs=4 total: nonzeros=746512, allocated nonzeros=746512 total number of mallocs used during MatSetValues calls=0 Up solver (post-smoother) same as down solver (pre-smoother) linear system matrix = precond matrix: Mat Object: 4 MPI processes type: mpiaij rows=37636, cols=37636, bs=4 total: nonzeros=746512, allocated nonzeros=746512 total number of mallocs used during MatSetValues calls=0 Number of SNES iterations = 2
Run on 1 processor with the default linear solver and profile the run
$ mpiexec -n 1 ./ex19 -da_refine 5 -log_view
Expected output:
lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./ex19 on a arch-ubuntu-mb-double-extra-opt named pdsbox with 1 processor, by patrick Fri Jul 27 15:30:21 2018 Using Petsc Development GIT revision: v3.9.3-921-gfc2aa81 GIT Date: 2018-07-27 11:07:58 +0200 Max Max/Min Avg Total Time (sec): 3.068e+00 1.000 3.068e+00 Objects: 9.400e+01 1.000 9.400e+01 Flops: 3.195e+09 1.000 3.195e+09 3.195e+09 Flops/sec: 1.041e+09 1.000 1.041e+09 1.041e+09 MPI Msg Count: 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Msg Len (bytes): 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Reductions: 0.000e+00 0.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 3.0680e+00 100.0% 3.1952e+09 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage SNESSolve 1 1.0 3.0142e+00 1.0 3.20e+09 1.0 0.0e+00 0.0e+00 0.0e+00 98100 0 0 0 98100 0 0 0 1060 SNESFunctionEval 45 1.0 4.6601e-02 1.0 3.56e+07 1.0 0.0e+00 0.0e+00 0.0e+00 2 1 0 0 0 2 1 0 0 0 763 SNESJacobianEval 2 1.0 8.7663e-02 1.0 3.63e+07 1.0 0.0e+00 0.0e+00 0.0e+00 3 1 0 0 0 3 1 0 0 0 414 SNESLineSearch 2 1.0 6.2666e-03 1.0 5.32e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 849 VecDot 2 1.0 7.8201e-05 1.0 1.51e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1925 VecMDot 568 1.0 2.6999e-01 1.0 6.61e+08 1.0 0.0e+00 0.0e+00 0.0e+00 9 21 0 0 0 9 21 0 0 0 2447 VecNorm 593 1.0 1.2326e-02 1.0 4.46e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 3621 VecScale 587 1.0 7.1690e-03 1.0 2.21e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 3082 VecCopy 63 1.0 1.7498e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 64 1.0 4.0205e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 76 1.0 2.5930e-03 1.0 5.72e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2206 VecWAXPY 2 1.0 2.6870e-04 1.0 7.53e+04 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 280 VecMAXPY 587 1.0 4.1431e-01 1.0 7.03e+08 1.0 0.0e+00 0.0e+00 0.0e+00 14 22 0 0 0 14 22 0 0 0 1698 VecScatterBegin 45 1.0 7.8702e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecReduceArith 4 1.0 9.0122e-05 1.0 3.01e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 3341 VecReduceComm 2 1.0 1.6689e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 587 1.0 2.0009e-02 1.0 6.63e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 3312 MatMult 587 1.0 8.6304e-01 1.0 8.54e+08 1.0 0.0e+00 0.0e+00 0.0e+00 28 27 0 0 0 28 27 0 0 0 990 MatSolve 587 1.0 1.2751e+00 1.0 8.54e+08 1.0 0.0e+00 0.0e+00 0.0e+00 42 27 0 0 0 42 27 0 0 0 670 MatLUFactorNum 2 1.0 5.5508e-02 1.0 1.41e+07 1.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 254 MatILUFactorSym 1 1.0 9.9506e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 3 1.0 1.4305e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 3 1.0 6.2709e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetRowIJ 1 1.0 4.0531e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 1.2467e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 2 1.0 4.0841e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatFDColorCreate 1 1.0 2.3603e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatFDColorSetUp 1 1.0 2.7331e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatFDColorApply 2 1.0 5.9345e-02 1.0 3.63e+07 1.0 0.0e+00 0.0e+00 0.0e+00 2 1 0 0 0 2 1 0 0 0 611 MatFDColorFunc 42 1.0 4.4162e-02 1.0 3.32e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 752 KSPSetUp 2 1.0 6.0439e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 2 1.0 2.9187e+00 1.0 3.15e+09 1.0 0.0e+00 0.0e+00 0.0e+00 95 99 0 0 0 95 99 0 0 0 1080 KSPGMRESOrthog 568 1.0 6.6268e-01 1.0 1.32e+09 1.0 0.0e+00 0.0e+00 0.0e+00 22 41 0 0 0 22 41 0 0 0 1994 PCSetUp 2 1.0 6.6768e-02 1.0 1.41e+07 1.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 211 PCApply 587 1.0 1.2759e+00 1.0 8.54e+08 1.0 0.0e+00 0.0e+00 0.0e+00 42 27 0 0 0 42 27 0 0 0 670 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage SNES 1 1 1372 0. DMSNES 1 1 672 0. SNESLineSearch 1 1 1000 0. Vector 47 47 13320304 0. Matrix 2 2 19277596 0. Matrix FD Coloring 1 1 16612048 0. Distributed Mesh 1 1 5344 0. Index Set 29 29 738052 0. IS L to G Mapping 2 2 189524 0. Star Forest Graph 2 2 1728 0. Discrete System 1 1 932 0. Vec Scatter 2 2 1408 0. Krylov Solver 1 1 18416 0. DMKSP interface 1 1 656 0. Preconditioner 1 1 1008 0. Viewer 1 0 0 0. ======================================================================================================================== Average time to get PetscTime(): 4.76837e-08 #PETSc Option Table entries: -da_refine 5 -log_view #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: PETSC_DIR=/home/patrick/petsc-mb PETSC_ARCH=arch-ubuntu-mb-double-extra-opt --with-debugging=0 --with-precision=double --with-scalar-type=real --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --COPTFLAGS="-g -O3 -march=native " --CXXOPTFLAGS="-g -O3 -march=native " --FOPTFLAGS="-g -O3 -march=native " --download-c2html --download-suitesparse --download-yaml --download-hdf5 --download-scalapack --download-metis --download-parmetis --download-mumps --download-superlu_dist --download-triangle --download-ctetgen --download-sundials --download-ml --download-exodusii --download-hdf5 --download-netcdf --download-pnetcdf ----------------------------------------- Libraries compiled on 2018-07-27 13:01:14 on pdsbox Machine characteristics: Linux-4.13.0-39-generic-x86_64-with-Ubuntu-16.04-xenial Using PETSc directory: /home/patrick/petsc-mb Using PETSc arch: arch-ubuntu-mb-double-extra-opt ----------------------------------------- Using C compiler: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden Using Fortran compiler: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpif90 -Wall -ffree-line-length-0 -Wno-unused-dummy-argument ----------------------------------------- Using include paths: -I/home/patrick/petsc-mb/include -I/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/include ----------------------------------------- Using C linker: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpicc Using Fortran linker: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpif90 Using libraries: -Wl,-rpath,/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -L/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -lpetsc -Wl,-rpath,/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -L/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/5 -L/usr/lib/gcc/x86_64-linux-gnu/5 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lpthread -lscalapack -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lsuperlu_dist -lml -lsundials_cvode -lsundials_nvecserial -lsundials_nvecparallel -llapack -lblas -lexodus -lnetcdf -lpnetcdf -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lparmetis -lmetis -ltriangle -lm -lctetgen -lpthread -lyaml -lstdc++ -ldl -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl -----------------------------------------
Search for the line beginning with SNESSolve, the fourth column gives the time for the nonlinear solve.
Run on 1 processor with the geometric multigrid linear solver and profile the run
$ mpiexec -n 1 ./ex19 -da_refine 5 -log_view -pc_type mg
Expected output:
lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. Number of SNES iterations = 3 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./ex19 on a arch-ubuntu-mb-double-extra-opt named pdsbox with 1 processor, by patrick Fri Jul 27 15:30:40 2018 Using Petsc Development GIT revision: v3.9.3-921-gfc2aa81 GIT Date: 2018-07-27 11:07:58 +0200 Max Max/Min Avg Total Time (sec): 6.992e-01 1.000 6.992e-01 Objects: 4.800e+02 1.000 4.800e+02 Flops: 5.237e+08 1.000 5.237e+08 5.237e+08 Flops/sec: 7.490e+08 1.000 7.490e+08 7.490e+08 MPI Msg Count: 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Msg Len (bytes): 0.000e+00 0.000 0.000e+00 0.000e+00 MPI Reductions: 0.000e+00 0.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 6.9923e-01 100.0% 5.2371e+08 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage SNESSolve 1 1.0 6.4513e-01 1.0 5.24e+08 1.0 0.0e+00 0.0e+00 0.0e+00 92100 0 0 0 92100 0 0 0 812 SNESFunctionEval 255 1.0 5.6777e-02 1.0 4.71e+07 1.0 0.0e+00 0.0e+00 0.0e+00 8 9 0 0 0 8 9 0 0 0 829 SNESJacobianEval 12 1.0 1.1569e-01 1.0 4.89e+07 1.0 0.0e+00 0.0e+00 0.0e+00 17 9 0 0 0 17 9 0 0 0 422 SNESLineSearch 2 1.0 5.7764e-03 1.0 5.32e+06 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 921 VecDot 2 1.0 7.6056e-05 1.0 1.51e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1979 VecMDot 111 1.0 7.1726e-03 1.0 1.38e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 1930 VecNorm 139 1.0 1.0304e-03 1.0 3.71e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 3600 VecScale 123 1.0 6.1989e-04 1.0 1.60e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2586 VecCopy 321 1.0 1.7195e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 382 1.0 4.5128e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecAXPY 252 1.0 3.0298e-03 1.0 4.41e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1455 VecAYPX 520 1.0 1.4517e-02 1.0 6.58e+06 1.0 0.0e+00 0.0e+00 0.0e+00 2 1 0 0 0 2 1 0 0 0 453 VecAXPBYCZ 260 1.0 9.2232e-03 1.0 1.32e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 1427 VecWAXPY 2 1.0 2.7275e-04 1.0 7.53e+04 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 276 VecMAXPY 123 1.0 1.3335e-02 1.0 1.67e+07 1.0 0.0e+00 0.0e+00 0.0e+00 2 3 0 0 0 2 3 0 0 0 1252 VecScatterBegin 265 1.0 1.1539e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecReduceArith 4 1.0 8.3208e-05 1.0 3.01e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 3618 VecReduceComm 2 1.0 1.4305e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 123 1.0 1.6305e-03 1.0 4.81e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 2949 MatMult 513 1.0 1.9088e-01 1.0 2.10e+08 1.0 0.0e+00 0.0e+00 0.0e+00 27 40 0 0 0 27 40 0 0 0 1100 MatMultAdd 65 1.0 5.0337e-03 1.0 2.93e+06 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 582 MatMultTranspose 70 1.0 5.1179e-03 1.0 3.16e+06 1.0 0.0e+00 0.0e+00 0.0e+00 1 1 0 0 0 1 1 0 0 0 617 MatSolve 13 1.0 1.2708e-04 1.0 4.91e+04 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 386 MatSOR 500 1.0 2.4601e-01 1.0 2.00e+08 1.0 0.0e+00 0.0e+00 0.0e+00 35 38 0 0 0 35 38 0 0 0 813 MatLUFactorSym 1 1.0 1.1539e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 2 1.0 1.2755e-04 1.0 1.82e+04 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 143 MatResidual 65 1.0 2.3766e-02 1.0 2.60e+07 1.0 0.0e+00 0.0e+00 0.0e+00 3 5 0 0 0 3 5 0 0 0 1094 MatAssemblyBegin 23 1.0 6.9141e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 23 1.0 8.9450e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatGetRowIJ 1 1.0 1.2636e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 4.2200e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 12 1.0 5.6696e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatFDColorCreate 6 1.0 2.8205e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatFDColorSetUp 6 1.0 3.7257e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 5 0 0 0 0 0 MatFDColorApply 12 1.0 7.7073e-02 1.0 4.89e+07 1.0 0.0e+00 0.0e+00 0.0e+00 11 9 0 0 0 11 9 0 0 0 634 MatFDColorFunc 252 1.0 5.5366e-02 1.0 4.47e+07 1.0 0.0e+00 0.0e+00 0.0e+00 8 9 0 0 0 8 9 0 0 0 807 DMCoarsen 5 1.0 2.9874e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 DMCreateInterp 5 1.0 3.7532e-03 1.0 2.25e+05 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 60 KSPSetUp 19 1.0 9.5367e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 KSPSolve 2 1.0 5.5405e-01 1.0 4.81e+08 1.0 0.0e+00 0.0e+00 0.0e+00 79 92 0 0 0 79 92 0 0 0 869 KSPGMRESOrthog 111 1.0 1.8478e-02 1.0 2.77e+07 1.0 0.0e+00 0.0e+00 0.0e+00 3 5 0 0 0 3 5 0 0 0 1498 PCSetUp 2 1.0 5.5341e-02 1.0 1.28e+07 1.0 0.0e+00 0.0e+00 0.0e+00 8 2 0 0 0 8 2 0 0 0 232 PCApply 13 1.0 4.7885e-01 1.0 4.45e+08 1.0 0.0e+00 0.0e+00 0.0e+00 68 85 0 0 0 68 85 0 0 0 928 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage SNES 1 1 1372 0. DMSNES 6 6 4432 0. SNESLineSearch 1 1 1000 0. Vector 206 206 18098080 0. Matrix 22 22 13706952 0. Matrix FD Coloring 6 6 22297904 0. Distributed Mesh 6 6 31664 0. Index Set 159 159 393244 0. IS L to G Mapping 12 12 261444 0. Star Forest Graph 12 12 9728 0. Discrete System 6 6 5572 0. Vec Scatter 17 17 11968 0. Krylov Solver 12 12 177272 0. DMKSP interface 6 6 3936 0. Preconditioner 7 7 6968 0. Viewer 1 0 0 0. ======================================================================================================================== Average time to get PetscTime(): 4.76837e-08 #PETSc Option Table entries: -da_refine 5 -log_view -pc_type mg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: PETSC_DIR=/home/patrick/petsc-mb PETSC_ARCH=arch-ubuntu-mb-double-extra-opt --with-debugging=0 --with-precision=double --with-scalar-type=real --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --COPTFLAGS="-g -O3 -march=native " --CXXOPTFLAGS="-g -O3 -march=native " --FOPTFLAGS="-g -O3 -march=native " --download-c2html --download-suitesparse --download-yaml --download-hdf5 --download-scalapack --download-metis --download-parmetis --download-mumps --download-superlu_dist --download-triangle --download-ctetgen --download-sundials --download-ml --download-exodusii --download-hdf5 --download-netcdf --download-pnetcdf ----------------------------------------- Libraries compiled on 2018-07-27 13:01:14 on pdsbox Machine characteristics: Linux-4.13.0-39-generic-x86_64-with-Ubuntu-16.04-xenial Using PETSc directory: /home/patrick/petsc-mb Using PETSc arch: arch-ubuntu-mb-double-extra-opt ----------------------------------------- Using C compiler: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden Using Fortran compiler: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpif90 -Wall -ffree-line-length-0 -Wno-unused-dummy-argument ----------------------------------------- Using include paths: -I/home/patrick/petsc-mb/include -I/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/include ----------------------------------------- Using C linker: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpicc Using Fortran linker: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpif90 Using libraries: -Wl,-rpath,/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -L/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -lpetsc -Wl,-rpath,/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -L/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/5 -L/usr/lib/gcc/x86_64-linux-gnu/5 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lpthread -lscalapack -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lsuperlu_dist -lml -lsundials_cvode -lsundials_nvecserial -lsundials_nvecparallel -llapack -lblas -lexodus -lnetcdf -lpnetcdf -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lparmetis -lmetis -ltriangle -lm -lctetgen -lpthread -lyaml -lstdc++ -ldl -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl -----------------------------------------
Compare the runtime for SNESSolve to the case with the default solver
Run on 4 processors with the default linear solver and profile the run
$ mpiexec -n 4 ./ex19 -da_refine 5 -log_view
Expected output:
lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. Number of SNES iterations = 2 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./ex19 on a arch-ubuntu-mb-double-extra-opt named pdsbox with 4 processors, by patrick Fri Jul 27 15:30:00 2018 Using Petsc Development GIT revision: v3.9.3-921-gfc2aa81 GIT Date: 2018-07-27 11:07:58 +0200 Max Max/Min Avg Total Time (sec): 1.200e+00 1.000 1.200e+00 Objects: 9.900e+01 1.000 9.900e+01 Flops: 9.349e+08 1.042 9.158e+08 3.663e+09 Flops/sec: 7.789e+08 1.042 7.631e+08 3.053e+09 MPI Msg Count: 1.453e+03 1.000 1.453e+03 5.811e+03 MPI Msg Len (bytes): 2.266e+06 1.021 1.544e+03 8.972e+06 MPI Reductions: 1.535e+03 1.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 1.2001e+00 100.0% 3.6633e+09 100.0% 5.811e+03 100.0% 1.544e+03 100.0% 1.528e+03 99.5% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSidedF 3 1.0 5.1808e-0314.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 SNESSolve 1 1.0 1.1392e+00 1.0 9.35e+08 1.0 5.8e+03 1.6e+03 1.5e+03 95100 99100 98 95100 99100 98 3216 SNESFunctionEval 45 1.0 6.7165e-03 1.0 9.08e+06 1.0 2.4e+01 1.6e+03 0.0e+00 1 1 0 0 0 1 1 0 0 0 5295 SNESJacobianEval 2 1.0 2.6000e-02 1.0 9.26e+06 1.0 3.4e+02 1.5e+03 8.6e+01 2 1 6 6 6 2 1 6 6 6 1395 SNESLineSearch 2 1.0 1.9200e-03 1.0 1.36e+06 1.0 3.2e+01 1.6e+03 8.0e+00 0 0 1 1 1 0 0 1 1 1 2771 VecDot 2 1.0 2.2244e-04 2.2 3.84e+04 1.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 677 VecMDot 655 1.0 2.0498e-01 1.1 1.94e+08 1.0 0.0e+00 0.0e+00 6.6e+02 16 21 0 0 43 16 21 0 0 43 3705 VecNorm 683 1.0 4.9419e-02 1.3 1.31e+07 1.0 0.0e+00 0.0e+00 6.8e+02 4 1 0 0 44 4 1 0 0 45 1040 VecScale 677 1.0 2.5268e-03 1.1 6.50e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 10084 VecCopy 66 1.0 5.6410e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 703 1.0 2.7184e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 82 1.0 5.7197e-04 1.1 1.58e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 10791 VecWAXPY 2 1.0 5.7220e-05 1.1 1.92e+04 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1315 VecMAXPY 677 1.0 1.2249e-01 1.1 2.06e+08 1.0 0.0e+00 0.0e+00 0.0e+00 10 22 0 0 0 10 22 0 0 0 6603 VecScatterBegin 722 1.0 9.7113e-03 1.1 0.00e+00 0.0 5.8e+03 1.6e+03 0.0e+00 1 0 99100 0 1 0 99100 0 0 VecScatterEnd 722 1.0 1.0391e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 VecReduceArith 4 1.0 4.0293e-05 1.1 7.68e+04 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 7472 VecReduceComm 2 1.0 7.3195e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 677 1.0 5.1487e-02 1.2 1.95e+07 1.0 0.0e+00 0.0e+00 6.8e+02 4 2 0 0 44 4 2 0 0 44 1485 MatMult 677 1.0 3.2710e-01 1.0 2.51e+08 1.0 5.4e+03 1.6e+03 0.0e+00 27 27 93 94 0 27 27 93 94 0 3012 MatSolve 677 1.0 3.9744e-01 1.0 2.49e+08 1.0 0.0e+00 0.0e+00 0.0e+00 33 27 0 0 0 33 27 0 0 0 2458 MatLUFactorNum 2 1.0 9.7592e-03 1.0 3.53e+06 1.1 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 1405 MatILUFactorSym 1 1.0 2.1026e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyBegin 3 1.0 5.2419e-0312.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAssemblyEnd 3 1.0 3.2048e-03 1.0 0.00e+00 0.0 1.6e+01 3.9e+02 8.0e+00 0 0 0 0 1 0 0 0 0 1 0 MatGetRowIJ 1 1.0 3.8147e-06 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 2.5654e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 2 1.0 1.5545e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatFDColorCreate 1 1.0 1.1539e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatFDColorSetUp 1 1.0 1.1923e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+01 1 0 0 0 5 1 0 0 0 5 0 MatFDColorApply 2 1.0 1.4742e-02 1.1 9.26e+06 1.0 3.4e+02 1.6e+03 3.0e+00 1 1 6 6 0 1 1 6 6 0 2461 MatFDColorFunc 42 1.0 8.3950e-03 1.2 8.47e+06 1.0 3.4e+02 1.6e+03 0.0e+00 1 1 6 6 0 1 1 6 6 0 3954 KSPSetUp 4 1.0 4.4918e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+01 0 0 0 0 1 0 0 0 0 1 0 KSPSolve 2 1.0 1.1108e+00 1.0 9.24e+08 1.0 5.4e+03 1.6e+03 1.4e+03 93 99 93 93 91 93 99 93 93 92 3260 KSPGMRESOrthog 655 1.0 3.2223e-01 1.1 3.88e+08 1.0 0.0e+00 0.0e+00 6.6e+02 26 41 0 0 43 26 41 0 0 43 4714 PCSetUp 4 1.0 1.2222e-02 1.0 3.53e+06 1.1 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 1122 PCSetUpOnBlocks 2 1.0 1.2162e-02 1.0 3.53e+06 1.1 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 1128 PCApply 677 1.0 4.1153e-01 1.0 2.49e+08 1.0 0.0e+00 0.0e+00 0.0e+00 34 27 0 0 0 34 27 0 0 0 2374 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage SNES 1 1 1372 0. DMSNES 1 1 672 0. SNESLineSearch 1 1 1000 0. Vector 49 49 3390624 0. Matrix 4 4 5105468 0. Matrix FD Coloring 1 1 4240080 0. Distributed Mesh 1 1 5344 0. Index Set 29 29 207404 0. IS L to G Mapping 1 1 10672 0. Star Forest Graph 2 2 1728 0. Discrete System 1 1 932 0. Vec Scatter 2 2 22184 0. Krylov Solver 2 2 19592 0. DMKSP interface 1 1 656 0. Preconditioner 2 2 1912 0. Viewer 1 0 0 0. ======================================================================================================================== Average time to get PetscTime(): 2.38419e-08 Average time for MPI_Barrier(): 2.91824e-05 Average time for zero size MPI_Send(): 8.88109e-06 #PETSc Option Table entries: -da_refine 5 -log_view #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: PETSC_DIR=/home/patrick/petsc-mb PETSC_ARCH=arch-ubuntu-mb-double-extra-opt --with-debugging=0 --with-precision=double --with-scalar-type=real --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --COPTFLAGS="-g -O3 -march=native " --CXXOPTFLAGS="-g -O3 -march=native " --FOPTFLAGS="-g -O3 -march=native " --download-c2html --download-suitesparse --download-yaml --download-hdf5 --download-scalapack --download-metis --download-parmetis --download-mumps --download-superlu_dist --download-triangle --download-ctetgen --download-sundials --download-ml --download-exodusii --download-hdf5 --download-netcdf --download-pnetcdf ----------------------------------------- Libraries compiled on 2018-07-27 13:01:14 on pdsbox Machine characteristics: Linux-4.13.0-39-generic-x86_64-with-Ubuntu-16.04-xenial Using PETSc directory: /home/patrick/petsc-mb Using PETSc arch: arch-ubuntu-mb-double-extra-opt ----------------------------------------- Using C compiler: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden Using Fortran compiler: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpif90 -Wall -ffree-line-length-0 -Wno-unused-dummy-argument ----------------------------------------- Using include paths: -I/home/patrick/petsc-mb/include -I/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/include ----------------------------------------- Using C linker: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpicc Using Fortran linker: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpif90 Using libraries: -Wl,-rpath,/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -L/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -lpetsc -Wl,-rpath,/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -L/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/5 -L/usr/lib/gcc/x86_64-linux-gnu/5 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lpthread -lscalapack -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lsuperlu_dist -lml -lsundials_cvode -lsundials_nvecserial -lsundials_nvecparallel -llapack -lblas -lexodus -lnetcdf -lpnetcdf -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lparmetis -lmetis -ltriangle -lm -lctetgen -lpthread -lyaml -lstdc++ -ldl -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl -----------------------------------------
Compare the runtime for
SNESSolveto the 1 processor case with the default solver. What is the speedup?Run on 4 processors with the geometric multigrid linear solver and profile the run
$ mpiexec -n 4 ./ex19 -da_refine 5 -log_view -pc_type mg
Expected output:
lid velocity = 0.000106281, prandtl # = 1., grashof # = 1. Number of SNES iterations = 3 ************************************************************************************************************************ *** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document *** ************************************************************************************************************************ ---------------------------------------------- PETSc Performance Summary: ---------------------------------------------- ./ex19 on a arch-ubuntu-mb-double-extra-opt named pdsbox with 4 processors, by patrick Fri Jul 27 15:29:45 2018 Using Petsc Development GIT revision: v3.9.3-921-gfc2aa81 GIT Date: 2018-07-27 11:07:58 +0200 Max Max/Min Avg Total Time (sec): 4.796e-01 1.001 4.794e-01 Objects: 5.730e+02 1.000 5.730e+02 Flops: 2.266e+08 1.061 2.201e+08 8.802e+08 Flops/sec: 4.726e+08 1.060 4.590e+08 1.836e+09 MPI Msg Count: 3.012e+03 1.043 2.950e+03 1.180e+04 MPI Msg Len (bytes): 1.664e+06 1.054 5.490e+02 6.478e+06 MPI Reductions: 1.472e+03 1.000 Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract) e.g., VecAXPY() for real vectors of length N --> 2N flops and VecAXPY() for complex vectors of length N --> 8N flops Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions -- Avg %Total Avg %Total Count %Total Avg %Total Count %Total 0: Main Stage: 4.7942e-01 100.0% 8.8021e+08 100.0% 1.180e+04 100.0% 5.490e+02 100.0% 1.465e+03 99.5% ------------------------------------------------------------------------------------------------------------------------ See the 'Profiling' chapter of the users' manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors) ------------------------------------------------------------------------------------------------------------------------ Event Count Time (sec) Flop --- Global --- --- Stage ---- Total Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s ------------------------------------------------------------------------------------------------------------------------ --- Event Stage 0: Main Stage BuildTwoSidedF 29 1.0 5.8196e-03 6.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 SNESSolve 1 1.0 4.1890e-01 1.0 2.27e+08 1.1 1.2e+04 5.5e+02 1.4e+03 87100100100 98 87100100100 98 2101 SNESFunctionEval 382 1.0 1.7268e-02 1.4 1.81e+07 1.1 3.2e+01 1.6e+03 0.0e+00 3 8 0 1 0 3 8 0 1 0 4066 SNESJacobianEval 18 1.0 6.6740e-02 1.0 1.89e+07 1.1 3.0e+03 5.2e+02 5.3e+02 14 8 26 24 36 14 8 26 24 36 1098 SNESLineSearch 3 1.0 2.8355e-03 1.0 2.04e+06 1.0 4.8e+01 1.6e+03 1.2e+01 1 1 0 1 1 1 1 0 1 1 2814 VecDot 3 1.0 2.3961e-04 1.3 5.76e+04 1.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 942 VecMDot 170 1.0 2.7825e-02 2.9 5.78e+06 1.1 0.0e+00 0.0e+00 1.7e+02 4 3 0 0 12 4 3 0 0 12 808 VecNorm 211 1.0 1.4362e-02 1.5 1.48e+06 1.1 0.0e+00 0.0e+00 2.1e+02 3 1 0 0 14 3 1 0 0 14 401 VecScale 188 1.0 4.2009e-04 1.2 6.51e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 6036 VecCopy 499 1.0 1.2398e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecSet 504 1.0 4.2868e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecAXPY 378 1.0 6.4588e-04 1.4 1.70e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 10235 VecAYPX 920 1.0 1.0263e-02 1.4 3.00e+06 1.1 0.0e+00 0.0e+00 0.0e+00 2 1 0 0 0 2 1 0 0 0 1134 VecAXPBYCZ 460 1.0 5.8384e-03 1.2 6.00e+06 1.1 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 3988 VecWAXPY 3 1.0 1.2279e-04 1.4 2.88e+04 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 920 VecMAXPY 188 1.0 7.0734e-03 1.4 6.95e+06 1.1 0.0e+00 0.0e+00 0.0e+00 1 3 0 0 0 1 3 0 0 0 3823 VecScatterBegin 1541 1.0 1.5262e-02 1.1 0.00e+00 0.0 1.1e+04 5.6e+02 0.0e+00 3 0 97 98 0 3 0 97 98 0 0 VecScatterEnd 1541 1.0 4.6712e-02 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 6 0 0 0 0 6 0 0 0 0 0 VecReduceArith 6 1.0 6.8903e-05 1.3 1.15e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 6554 VecReduceComm 3 1.0 2.2459e-04 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0 VecNormalize 188 1.0 1.3350e-02 1.6 1.95e+06 1.1 0.0e+00 0.0e+00 1.9e+02 2 1 0 0 13 2 1 0 0 13 570 MatMult 878 1.0 1.2342e-01 1.2 9.29e+07 1.1 6.9e+03 6.4e+02 0.0e+00 23 41 59 68 0 23 41 59 68 0 2925 MatMultAdd 115 1.0 4.8120e-03 1.1 1.32e+06 1.0 5.8e+02 2.6e+02 0.0e+00 1 1 5 2 0 1 1 5 2 0 1078 MatMultTranspose 120 1.0 1.4319e-02 3.1 1.38e+06 1.0 6.0e+02 2.6e+02 0.0e+00 2 1 5 2 0 2 1 5 2 0 378 MatSolve 23 1.0 2.2483e-04 1.1 8.68e+04 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1545 MatSOR 855 1.0 1.5684e-01 1.3 8.70e+07 1.1 0.0e+00 0.0e+00 0.0e+00 28 38 0 0 0 28 38 0 0 0 2154 MatLUFactorSym 1 1.0 1.2636e-04 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatLUFactorNum 3 1.0 1.7381e-04 1.2 2.73e+04 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 628 MatCopy 2 1.0 3.3379e-06 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatConvert 1 1.0 3.6716e-05 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatResidual 115 1.0 1.8372e-02 1.4 1.19e+07 1.1 9.2e+02 6.1e+02 0.0e+00 3 5 8 9 0 3 5 8 9 0 2504 MatAssemblyBegin 30 1.0 6.0785e-03 5.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 MatAssemblyEnd 30 1.0 8.9223e-03 1.1 0.00e+00 0.0 1.5e+02 9.3e+01 8.8e+01 2 0 1 0 6 2 0 1 0 6 0 MatGetRowIJ 1 1.0 8.3447e-06 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatCreateSubMats 3 1.0 2.5177e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatGetOrdering 1 1.0 3.4571e-05 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatZeroEntries 18 1.0 4.1199e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatFDColorCreate 6 1.0 4.7135e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 1 0 0 0 0 1 0 MatFDColorSetUp 6 1.0 2.7599e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 4.8e+02 6 0 0 0 33 6 0 0 0 33 0 MatFDColorApply 18 1.0 3.8840e-02 1.0 1.89e+07 1.1 3.0e+03 5.2e+02 2.8e+01 8 8 26 24 2 8 8 26 24 2 1887 MatFDColorFunc 378 1.0 2.6123e-02 1.1 1.73e+07 1.1 3.0e+03 5.2e+02 0.0e+00 5 8 26 24 0 5 8 26 24 0 2566 MatRedundantMat 3 1.0 3.0828e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatMPIConcateSeq 3 1.0 4.4107e-05 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 DMCoarsen 5 1.0 1.9548e-03 1.0 0.00e+00 0.0 8.0e+01 2.2e+01 5.5e+01 0 0 1 0 4 0 0 1 0 4 0 DMCreateInterp 5 1.0 6.3477e-03 1.0 5.76e+04 1.0 1.2e+02 6.6e+01 1.2e+02 1 0 1 0 8 1 0 1 0 8 36 KSPSetUp 29 1.0 3.4931e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 7.6e+01 1 0 0 0 5 1 0 0 0 5 0 KSPSolve 3 1.0 3.7720e-01 1.0 2.10e+08 1.1 1.1e+04 5.0e+02 1.3e+03 79 93 95 86 91 79 93 95 86 91 2166 KSPGMRESOrthog 170 1.0 3.3011e-02 2.1 1.16e+07 1.1 0.0e+00 0.0e+00 1.7e+02 5 5 0 0 12 5 5 0 0 12 1363 PCSetUp 3 1.0 5.2358e-02 1.0 5.09e+06 1.1 2.9e+03 3.2e+02 8.2e+02 11 2 25 14 56 11 2 25 14 56 367 PCApply 23 1.0 3.0923e-01 1.0 1.94e+08 1.1 8.1e+03 5.4e+02 4.4e+02 63 86 69 69 30 63 86 69 69 30 2434 ------------------------------------------------------------------------------------------------------------------------ Memory usage is given in bytes: Object Type Creations Destructions Memory Descendants' Mem. Reports information only for process 0. --- Event Stage 0: Main Stage SNES 1 1 1372 0. DMSNES 6 6 4432 0. SNESLineSearch 1 1 1000 0. Vector 231 231 4858432 0. Matrix 56 56 4056840 0. Matrix FD Coloring 6 6 5749152 0. Distributed Mesh 6 6 31664 0. Index Set 185 185 220700 0. IS L to G Mapping 6 6 17912 0. Star Forest Graph 12 12 9728 0. Discrete System 6 6 5572 0. Vec Scatter 29 29 97136 0. Krylov Solver 13 13 178448 0. DMKSP interface 6 6 3936 0. Preconditioner 8 8 7904 0. Viewer 1 0 0 0. ======================================================================================================================== Average time to get PetscTime(): 4.76837e-08 Average time for MPI_Barrier(): 2.79903e-05 Average time for zero size MPI_Send(): 1.04904e-05 #PETSc Option Table entries: -da_refine 5 -log_view -pc_type mg #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: PETSC_DIR=/home/patrick/petsc-mb PETSC_ARCH=arch-ubuntu-mb-double-extra-opt --with-debugging=0 --with-precision=double --with-scalar-type=real --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --COPTFLAGS="-g -O3 -march=native " --CXXOPTFLAGS="-g -O3 -march=native " --FOPTFLAGS="-g -O3 -march=native " --download-c2html --download-suitesparse --download-yaml --download-hdf5 --download-scalapack --download-metis --download-parmetis --download-mumps --download-superlu_dist --download-triangle --download-ctetgen --download-sundials --download-ml --download-exodusii --download-hdf5 --download-netcdf --download-pnetcdf ----------------------------------------- Libraries compiled on 2018-07-27 13:01:14 on pdsbox Machine characteristics: Linux-4.13.0-39-generic-x86_64-with-Ubuntu-16.04-xenial Using PETSc directory: /home/patrick/petsc-mb Using PETSc arch: arch-ubuntu-mb-double-extra-opt ----------------------------------------- Using C compiler: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpicc -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -fvisibility=hidden Using Fortran compiler: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpif90 -Wall -ffree-line-length-0 -Wno-unused-dummy-argument ----------------------------------------- Using include paths: -I/home/patrick/petsc-mb/include -I/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/include ----------------------------------------- Using C linker: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpicc Using Fortran linker: /home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/bin/mpif90 Using libraries: -Wl,-rpath,/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -L/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -lpetsc -Wl,-rpath,/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -L/home/patrick/petsc-mb/arch-ubuntu-mb-double-extra-opt/lib -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/5 -L/usr/lib/gcc/x86_64-linux-gnu/5 -Wl,-rpath,/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu -Wl,-rpath,/lib/x86_64-linux-gnu -L/lib/x86_64-linux-gnu -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lpthread -lscalapack -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lsuperlu_dist -lml -lsundials_cvode -lsundials_nvecserial -lsundials_nvecparallel -llapack -lblas -lexodus -lnetcdf -lpnetcdf -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lparmetis -lmetis -ltriangle -lm -lctetgen -lpthread -lyaml -lstdc++ -ldl -lmpifort -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl -----------------------------------------
Compare the runtime for SNESSolve to the 1 processor case with multigrid. What is the speedup? Why is the speedup for multigrid lower than the speedup for the default solver?
Nonlinear time dependent PDE on unstructured grid#
WHAT THIS EXAMPLE DEMONSTRATES:
Changing the default ODE integrator
Handling unstructured grids
Registering your own interchangeable physics and algorithm modules
FURTHER DETAILS:
DO THE FOLLOWING:
Compile
src/ts/tutorials/ex11.c$ cd petsc/src/ts/tutorials $ make ex11
Run simple advection through a tiny hybrid mesh
$ mpiexec -n 1 ./ex11 -f ${PETSC_DIR}/share/petsc/datafiles/meshes/sevenside.exo
Expected output:
0 time 0 |x| 7.155 1 time 0.7211 |x| 4.386 2 time 1.442 |x| 1.968 3 time 2.163 |x| 1.925 CONVERGED_TIME at time 2.16333 after 3 steps
Run simple advection through a small mesh with a Rosenbrock-W solver
$ mpiexec -n 1 ./ex11 -f ${PETSC_DIR}/share/petsc/datafiles/meshes/sevenside.exo -ts_type rosw
Expected output:
0 time 0 |x| 7.155 1 time 0.03915 |x| 7.035 2 time 0.1105 |x| 6.802 3 time 0.1895 |x| 6.525 4 time 0.2751 |x| 6.21 5 time 0.366 |x| 5.862 6 time 0.4606 |x| 5.492 7 time 0.5569 |x| 5.113 8 time 0.6526 |x| 4.736 9 time 0.7492 |x| 4.361 10 time 0.8467 |x| 3.99 11 time 0.9462 |x| 3.622 12 time 1.051 |x| 3.248 13 time 1.166 |x| 2.853 14 time 1.293 |x| 2.444 15 time 1.431 |x| 2.024 16 time 1.581 |x| 1.915 17 time 1.741 |x| 1.939 18 time 1.908 |x| 1.956 19 time 2.075 |x| 1.969 CONVERGED_TIME at time 2.07496 after 19 steps
Run simple advection through a larger quadrilateral mesh of an annulus with least squares reconstruction and no limiting, monitoring the error
$ mpiexec -n 4 ./ex11 -f ${PETSC_DIR}/share/petsc/datafiles/meshes/annulus-20.exo -monitor Error -advect_sol_type bump -petscfv_type leastsquares -petsclimiter_type sin
Expected output:
0 time 0 |x| 0.8708 Error [ 0, 0] int 0 1 time 0.07526 |x| 0.7882 Error [ 0,0.09434159] int 0.05288794 2 time 0.1505 |x| 0.7343 Error [ 0, 0.2131413] int 0.09120994 3 time 0.2258 |x| 0.6957 Error [ 0, 0.2529671] int 0.1178105 4 time 0.301 |x| 0.6728 Error [ 0, 0.2209969] int 0.1510195 5 time 0.3763 |x| 0.6484 Error [ 0, 0.3418217] int 0.1742279 6 time 0.4516 |x| 0.6259 Error [ 0, 0.2836391] int 0.2025824 7 time 0.5268 |x| 0.6001 Error [ 0, 0.2995109] int 0.2355702 8 time 0.6021 |x| 0.5671 Error [4.018435e-45, 0.4150825] int 0.2669753 9 time 0.6774 |x| 0.5406 Error [1.972482e-39, 0.3607147] int 0.2858846 10 time 0.7526 |x| 0.5171 Error [6.083192e-36, 0.448034] int 0.3098002 11 time 0.8279 |x| 0.506 Error [3.572914e-33, 0.4455137] int 0.3356554 12 time 0.9031 |x| 0.4947 Error [3.404038e-31, 0.4161921] int 0.3553281 13 time 0.9784 |x| 0.4851 Error [2.737647e-29, 0.3831727] int 0.3701208 14 time 1.054 |x| 0.4746 Error [1.595143e-27, 0.4870963] int 0.386669 15 time 1.129 |x| 0.4618 Error [4.855084e-26, 0.4884613] int 0.4033437 16 time 1.204 |x| 0.4501 Error [8.826408e-25, 0.4241629] int 0.4207896 17 time 1.279 |x| 0.4399 Error [1.013076e-23, 0.4846543] int 0.4351298 18 time 1.355 |x| 0.4309 Error [8.357122e-23, 0.5166439] int 0.4563886 19 time 1.43 |x| 0.4192 Error [5.938047e-22, 0.4947022] int 0.4741966 20 time 1.505 |x| 0.412 Error [3.987736e-21, 0.4909573] int 0.4931484 21 time 1.581 |x| 0.4017 Error [2.328432e-20, 0.5286506] int 0.5038741 22 time 1.656 |x| 0.3916 Error [1.188744e-19, 0.518947] int 0.5228681 23 time 1.731 |x| 0.3824 Error [5.375705e-19, 0.5598767] int 0.5402542 24 time 1.806 |x| 0.3774 Error [2.397704e-18, 0.5997036] int 0.5577958 25 time 1.882 |x| 0.3712 Error [1.010082e-17, 0.5943566] int 0.5687406 26 time 1.957 |x| 0.3641 Error [3.837194e-17, 0.5663827] int 0.5766598 27 time 2.032 |x| 0.3569 Error [1.311105e-16, 0.5534062] int 0.5847664 CONVERGED_TIME at time 2.03208 after 27 steps
Compare turning to the error after turning off reconstruction.
Run shallow water on the larger mesh with least squares reconstruction and minmod limiting, monitoring water Height (integral is conserved) and Energy (not conserved)
$ mpiexec -n 4 ./ex11 -f ${PETSC_DIR}/share/petsc/datafiles/meshes/annulus-20.exo -physics sw -monitor Height,Energy -petscfv_type leastsquares -petsclimiter_type minmod
Expected output:
0 time 0 |x| 2.972 Height [ 1, 2.97161] int 28.12971 Energy [ 0.5, 4.415232] int 17.12186 1 time 0.1597 |x| 2.125 Height [ 1, 2.125435] int 28.12971 Energy [ 0.5, 2.334901] int 17.2733 2 time 0.3193 |x| 1.944 Height [ 1, 1.943793] int 28.12971 Energy [ 0.5, 1.904131] int 17.29508 3 time 0.479 |x| 1.866 Height [ 1, 1.865932] int 28.12971 Energy [ 0.5, 1.741459] int 17.11151 4 time 0.6386 |x| 1.677 Height [ 1, 1.676699] int 28.12971 Energy [ 0.5, 1.420746] int 16.96103 5 time 0.7983 |x| 1.579 Height [ 1, 1.578751] int 28.12971 Energy [ 0.5, 1.266909] int 16.8442 6 time 0.9579 |x| 1.534 Height [ 0.9066044, 1.533904] int 28.12971 Energy [ 0.4148315, 1.211645] int 16.73017 7 time 1.118 |x| 1.522 Height [ 0.8458232, 1.5217] int 28.12971 Energy [ 0.3590109, 1.171816] int 16.64452 8 time 1.277 |x| 1.495 Height [ 0.8767922, 1.494585] int 28.12971 Energy [ 0.3846352, 1.132207] int 16.59414 9 time 1.437 |x| 1.462 Height [ 0.9860658, 1.46217] int 28.12971 Energy [ 0.486527, 1.084667] int 16.56137 10 time 1.597 |x| 1.424 Height [ 1.000033, 1.424019] int 28.12971 Energy [ 0.5000325, 1.024201] int 16.51911 11 time 1.756 |x| 1.381 Height [ 1.000129, 1.381206] int 28.12971 Energy [ 0.5001287, 0.9675924] int 16.46646 12 time 1.916 |x| 1.336 Height [ 1.000441, 1.336399] int 28.12971 Energy [ 0.5004409, 0.9293328] int 16.41464 13 time 2.076 |x| 1.308 Height [ 0.9827839, 1.30794] int 28.12971 Energy [ 0.4835028, 0.8976772] int 16.36243 CONVERGED_TIME at time 2.07552 after 13 steps