Several issues come to mind and hopefully an explanation of them will lay some of your fears to rest. Drilling pen blanks on a lathe is certainly a common function, but it does come with some risk, which you have already experienced. When drilling into the end of a pen blank you are drilling into end grain - and that can be tricky, especially if you are drilling a hole the entire length of the pen. Because of the long distance and the type of woods usually used for pen blanks (exotic or highly figured wood), there is a greater chance that the drill bit will wobble, or that the grain of the wood will cause the drill bit to stray off cnter, and the next thing you know you have some side (lateral) pressure which can knock the wood out of the chuck in a hurry. Scroll chucks are really not designed to hold square pieces of wood very well, especially small blanks like pens. A drill press with a clamp works much better. If you must use a lathe, then make sure the blank is as square as possible, and that, if possible, the end of the blank can rest on the inside of the chuck. Another option is to turn the blank round first, between centers, and then mount the rounded wood into the chuck for a better grip. The problem here is that pen blanks are usually small to begin with, and so rounding it first makes it even smaller, and thus your drilling has to be that much more accurate.
Weed pots present a similar challenge in that you are usually turning the wood like a spindle, in other words, you are making the hole for the pot through the endgrain, and this can put a lot of pressure on the wood if the drill or the gouge starts to go off center. For these items I would plan on wasting a bit of wood so that after you have turned the weed pot round between centers, you then make a tenon long enough to ensure that the end of the cylinder sits on the base of the chuck so that it will minimize side to side movement. This should ensure that the wood stays in the chuck. When turning bowls the same principle applies. Always try to make sure that the tenon is long enough to sit on the base of the chuck, at least until you get more comfortable with mounting wood in a chuck. It is better to be safer than not. With experience you will learn the best mounting methods for whatever you are turning, but for now, don't worry about maximizing your wood. Waste a little bit extra length to ensure a snug fit.
Also, look to see if the jaws on your chuck are cut at a light angle (also known as a dovetail). If they are, then by forming the tenon with a slight angle toward the base of your bowl or cylinder, you will ensure a tighter/stronger grip. In other words, if you were to look at the bottom of the tenon, it would be slightly wider at the base and taper slightly toward the bottom of your workpiece.
Just as an aside, even when using your chuck always use the tailstock to keep pressure on the wood for as long as possible. When you need to drill the end or hollow out the center, then of course you have to pull the tailstock away; but until then, use it to help further stabilize the workpiece and provide a margin of safety from flying wood.
Good luck - and hang in there!
"Keep those shavings flying!"