They're both ideal for tiles that are larger than 4 inches and most floor tiles. You'll get slightly more adhesive with square-notched trowels, but the ridges created by U-notch trowels typically ...

Spread a thin, even layer of VCT flooring adhesive with a 1/16-in. x 1/16-in. notched trowel. Spread glue over half the floor, starting at the wall and working up to the center layout line.

What caused the trowel marks? That’s impossible to say without pulling up at least a few planks. The bulges could be remnants of adhesive used to hold down an earlier glued-on floor covering.