That does sound wildly inconsistent. Try the following (apologies if this is teaching you to suck eggs)
Is the top hopper turned fully so the port is fully open to allow as much powder as poss to flow through?
Always ensure the hopper is at least half full of powder even if you are only throwing less charges than this amount justifies. It works better with a weight of powder on top.
Are you losing any kernels when dropping a charge? As usual with Lee stuff, they work but are not exactly designed perfectly. For example, you can charge directly to a case easily enough (which is probably how it was designed) but if you drop a charge in to a pan for example, it is hard to not have powder jump out. A couple of small kernels will be 1/10 of a grain, so you do not need to lose many to give variances. Try dropping directly in to a case (with a primer in

) and weighing those to see how consistent they are.
Consistency on technique. Sounds daft but flip the lever with the same force and allow the same time between up and down. I personally drop the charge with a clunk to ensure lever is deffo at botto of its travel, wait a couple of seconds and then move the lever back up and do whatever I am doing with the charged case before doing the next one. This gives time for the weight above to fully drop the entire volume back in to the tube. Althought the unit is made from materials that would fracture in a slight zepher, it can take a bit of knocking. Be positive with it.
I am not a massive fan of Lee stuff to be honest but I do actually rate the Lee powder thrower even with extruded powders. I get around +/-0.3grains variance but more often than not much much better. It is the odd drop that is off. I load with N130, N140 and N160 and all meter ok in my view. I use beam scales to set the thrower for ammo for deer and chuck loads direct to cases every second. It is fast and more than adequate for MOA accuracy out to 300yds for large targets like deer, as long as you have worked up an optimum charge weight (this should be done using beam scales only for absolute accuracy)
For almost all reloading, beam scales set up properly and a well sorted thrower are all you need.
Good luck.