One thing to check is that either one of the action screws (especially the front) is not interfering with the bolt. After awhile, the screws may seat down in the bottom metal or pillar(s), and touch the bolt. When this happens to the front screw, it can tilt the floating bolt head and toss a shot, depending on how the case chambers. I had this happen to a Savage Target Action. The screw just barely touched, I couldn't feel it and the bolt closed normally. There was just a slight mark on the locking lug.
Another cause can be an internal box magazine not properly seated in the action, where there is interference between the action and stock (the action is tightening against the magazine box rather than the stock). If yours is a top loader that has the Magazine Retaining Clip, #7 in the parts drawing, that retains the magazine box, be sure it is properly seated in the slot on the box and not distorted.
Finally, check that the action screw torque is set to factory recommendations. Also check that there isn't too much pressure between the rear tang and the stock, many Savages prefer minimal pressure reward of the action screw nearest the trigger.
Regarding your comment about seeing the POA shifting when dry firing. Usually this is normal - we have all seen our aim knocked off by muzzle blast from nearby shooters at the range. There are two things that may relate. If the cocking pin on the firing pin assembly hits the bolt body before the firing pin hits its stop, it can play havoc with firing pin protrusion and kink the bolt sideways. This may toss a shot up to the left as the cocking pin on the Savage travels on the right side of the action, not the bottom as in the Remington and Mausers, etc. There should be clearance between the cocking pin and the bolt body (in the cam area) when the pin is in the fired position. See
http://www.switchbarrel.com/Spring.htm read down to the second to the last paragraph. Also, firing pin protrusion should be somewhere between .040" to .060". The second is be sure the scope rings aren't tightened around the scope too much, as this will stress the erector tube and cause it to jam or bind. All that is needed is 15-18 inch pounds on the screws. Another factor is that the rings need to be inline with each other, or stress can be introduced. In some cases that requires either Burris Signature rings with the bushings, or a Farrell type one piece mount glass bedded to the action contour.
If there is too much oil and gunk in the bolt it can cause problems with function leading to ignition problems in cold weather. Dismantle the bolt, clean with solvent and lightly lubricate.