We install the AU and VST plugins in the user’s Library folder NOT the system Library folder.
The AU plugin is at location:
And the VSTs are at locations:
The AAX plugin is delivered as a disk image. Double click this to open it, then drag the plugin (on the left) into the folder (on the right). It will then end up at location:
If you need to open these folders use Finder's "Go" menu and choose "Go to folder…" then type in the paths as written above. Alternatively you can use Finder's "Go" and press alt ⌥. This will show you the 'hidden' user library, and you can navigate to the right folder from there.