It can sometimes be difficult to decide what sort of modifier to assign to certain tasks. The guidelines for trait tests in the core rules recommend +2 for an easy task, -2 for a difficult task, and -4 for a very difficult task, but I find it's also useful to look at existing modifiers to get a better feel for when and where they're used.
For example, when attacking someone there is a -1 penalty for dim lighting, -2 for dark lighting, -4 for pitch darkness, and -6 for complete darkness. Similarly, there's a -4 penalty for attacking someone who is only visible as a vague outline, and a -6 penalty for attacking a completely invisible foe.
Likewise when it comes to cover, there's a -1 penalty for attacking someone in light cover, -2 for medium cover, -4 for heavy cover, and -6 for near total cover. And with Called Shots, there's a -2 penalty for targeting a limb, -4 for the head or vitals, and -6 for tiny targets, such as the eye slit in a helmet. Conversely, there's a +1 bonus for tracking someone through a dusty area, +2 for tracking them through mud, and +4 for tracking them through fresh snow.
You'll also see -2 penalties for things like using your off hand, performing a task without appropriate tools, shooting from horseback, suffering recoil from an automatic weapon, performing two actions the same round, and so on. Meanwhile, many Professional Edges (which "represent many years of training") provide a +2 bonus to certain skill rolls.
For combat modifiers, using an improvised weapon incurs a -1 penalty to attack and Parry, while the Defend maneuver (focusing on pure defense) grants +2 Parry. Wild Attack represent an all-out offence, and grants +2 to attack and damage, but -2 Parry. The Drop represents catching someone completely off-guard, or trying to kill someone you're holding in a classic hostage pose, and grants +4 to attack and damage.
Based on the above observations, my personal guideline is to use one of four possible penalties: -1 or -2 for difficult tasks, -4 for very difficult tasks, and -6 for exceptionally difficult tasks.
In most cases I stick to -2 for difficult tasks and -4 for very difficult tasks, but sometimes I will assign a -1 penalty (for tasks that are noticeably more difficult than average, but clearly not as difficult as those listed as incurring a -2 penalty), and in extreme cases I will assign a -6 penalty.
For example, the Climbing Modifiers suggest a -2 penalty for "scarce or thin handholds", but I might reduce the penalty to -1 or increase it to -4, depending on just how scarce or thin the handholds are, increasing the penalty to -6 if there are no handholds at all (assuming the climb is even possible - but the point is, I wouldn't increase the penalty beyond -6).
The same approach can be used for bonuses. Antique climbing gear grants a +2 bonus to Climbing rolls, while modern gear grants a +4 bonus, so I might award a +1 bonus for improvised climbing gear and possibly a +6 bonus for high-tech futuristic climbing gear (with a huge caveat: it's usually better to have gear eliminate penalties rather than assign bonuses if you still want standard difficulty tasks to pose a challenge).
Of course it's also very important to stress that not all tasks require a bonus or penalty - most of the time it's perfectly sufficient to call for an unmodified roll!